[==========] Running 23 tests from 3 test cases. [----------] Global test environment set-up. [----------] 13 tests from RaftConsensusNonVoterITest [ RUN ] RaftConsensusNonVoterITest.GetTableAndTabletLocations WARNING: Logging before InitGoogleLogging() is written to STDERR I0507 06:28:20.429571 231 test_util.cc:212] Using random seed: 1387787996 I0507 06:28:20.441535 231 ts_itest-base.cc:112] Starting cluster with: I0507 06:28:20.441596 231 ts_itest-base.cc:113] -------------- I0507 06:28:20.441627 231 ts_itest-base.cc:114] 4 tablet servers I0507 06:28:20.441654 231 ts_itest-base.cc:115] 3 replicas per TS I0507 06:28:20.441684 231 ts_itest-base.cc:116] -------------- I0507 06:28:20.443996 231 external_mini_cluster.cc:933] Running /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-master /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-master --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.GetTableAndTabletLocations.1557210500371863-231/raft_consensus-itest-cluster/master-0/wal --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.GetTableAndTabletLocations.1557210500371863-231/raft_consensus-itest-cluster/master-0/data --block_manager=log --webserver_interface=localhost --ipki_ca_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.0.57.254:42455 --webserver_interface=127.0.57.254 --webserver_port=0 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.GetTableAndTabletLocations.1557210500371863-231/raft_consensus-itest-cluster/master-0/logs --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.GetTableAndTabletLocations.1557210500371863-231/raft_consensus-itest-cluster/master-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --catalog_manager_evict_excess_replicas=false --rpc_reuseport=true with env {} WARNING: Logging before InitGoogleLogging() is written to STDERR W0507 06:28:20.646281 241 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0507 06:28:20.646533 241 flags.cc:404] Enabled unsafe flag: --never_fsync=true W0507 06:28:20.649240 241 flags.cc:404] Enabled experimental flag: --ipki_ca_key_size=1024 W0507 06:28:20.649286 241 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W0507 06:28:20.649317 241 flags.cc:404] Enabled experimental flag: --tsk_num_rsa_bits=512 W0507 06:28:20.649389 241 flags.cc:404] Enabled experimental flag: --rpc_reuseport=true I0507 06:28:20.654062 241 master_main.cc:73] Master server non-default flags: --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.GetTableAndTabletLocations.1557210500371863-231/raft_consensus-itest-cluster/master-0/data --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.GetTableAndTabletLocations.1557210500371863-231/raft_consensus-itest-cluster/master-0/wal --catalog_manager_evict_excess_replicas=false --ipki_ca_key_size=1024 --ipki_server_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.0.57.254:42455 --rpc_reuseport=true --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.GetTableAndTabletLocations.1557210500371863-231/raft_consensus-itest-cluster/master-0/data/info.pb --webserver_interface=127.0.57.254 --webserver_port=0 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.GetTableAndTabletLocations.1557210500371863-231/raft_consensus-itest-cluster/master-0/logs --logbuflevel=-1 --logtostderr=true Master server version: kudu 1.10.0-cdh6.x-SNAPSHOT revision 7d0ee26697926e87af9ef395008799577f6ab43f build type FASTDEBUG built by jenkins at 06 May 2019 22:44:21 PST on quasar-knmpfx-1.vpc.cloudera.com build id 322017 ASAN enabled I0507 06:28:20.660918 241 system_ntp.cc:202] NTP initialized. Skew: 500ppm Current error: 24516us I0507 06:28:20.662964 241 fs_manager.cc:263] Metadata directory not provided I0507 06:28:20.663044 241 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0507 06:28:20.663321 241 server_base.cc:433] Could not load existing FS layout: Not found: could not find a healthy instance file I0507 06:28:20.663399 241 server_base.cc:434] Attempting to create new FS layout instead I0507 06:28:20.669096 241 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.GetTableAndTabletLocations.1557210500371863-231/raft_consensus-itest-cluster/master-0/data/instance: uuid: "2f68b13a6f104433814b4b2165a22e0e" format_stamp: "Formatted at 2019-05-07 06:28:20 on dist-test-slave-dist-test-slave-rqp6" I0507 06:28:20.669674 241 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.GetTableAndTabletLocations.1557210500371863-231/raft_consensus-itest-cluster/master-0/wal/instance: uuid: "2f68b13a6f104433814b4b2165a22e0e" format_stamp: "Formatted at 2019-05-07 06:28:20 on dist-test-slave-dist-test-slave-rqp6" I0507 06:28:20.676803 241 fs_manager.cc:503] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s I0507 06:28:20.677974 241 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0507 06:28:20.678581 241 file_cache.cc:466] Constructed file cache lbm with capacity 419430 I0507 06:28:20.682623 241 fs_manager.cc:419] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s I0507 06:28:20.682708 241 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.GetTableAndTabletLocations.1557210500371863-231/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.GetTableAndTabletLocations.1557210500371863-231/raft_consensus-itest-cluster/master-0/wal uuid: "2f68b13a6f104433814b4b2165a22e0e" format_stamp: "Formatted at 2019-05-07 06:28:20 on dist-test-slave-dist-test-slave-rqp6" I0507 06:28:20.682893 241 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.GetTableAndTabletLocations.1557210500371863-231/raft_consensus-itest-cluster/master-0/wal metadata directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.GetTableAndTabletLocations.1557210500371863-231/raft_consensus-itest-cluster/master-0/wal 1 data directories: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.GetTableAndTabletLocations.1557210500371863-231/raft_consensus-itest-cluster/master-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0507 06:28:20.716606 241 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0507 06:28:20.732436 241 rpc_server.cc:205] RPC server started. Bound to: 127.0.57.254:42455 I0507 06:28:20.733618 241 webserver.cc:308] Webserver started at http://127.0.57.254:43125/ using document root and password file I0507 06:28:20.734351 241 server_base.cc:608] Dumped server information to /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.GetTableAndTabletLocations.1557210500371863-231/raft_consensus-itest-cluster/master-0/data/info.pb I0507 06:28:20.739126 300 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0507 06:28:20.743924 231 external_mini_cluster.cc:995] Started /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-master as pid 241 I0507 06:28:20.744891 231 external_mini_cluster.cc:933] Running /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.GetTableAndTabletLocations.1557210500371863-231/raft_consensus-itest-cluster/ts-0/wal --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.GetTableAndTabletLocations.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data --block_manager=log --rpc_bind_addresses=127.0.57.193:0 --local_ip_for_outbound_sockets=127.0.57.193 --webserver_interface=127.0.57.193 --webserver_port=0 --tserver_master_addrs=127.0.57.254:42455 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.GetTableAndTabletLocations.1557210500371863-231/raft_consensus-itest-cluster/ts-0/logs --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.GetTableAndTabletLocations.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --log_cache_size_limit_mb=10 --consensus_rpc_timeout_ms=50 with env {} I0507 06:28:20.755471 300 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 2f68b13a6f104433814b4b2165a22e0e: Bootstrap starting. I0507 06:28:20.762529 300 tablet_bootstrap.cc:590] T 00000000000000000000000000000000 P 2f68b13a6f104433814b4b2165a22e0e: No blocks or log segments found. Creating new log. I0507 06:28:20.763500 300 log.cc:525] T 00000000000000000000000000000000 P 2f68b13a6f104433814b4b2165a22e0e: Log is configured to *not* fsync() on all Append() calls I0507 06:28:20.770081 300 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 2f68b13a6f104433814b4b2165a22e0e: No bootstrap required, opened a new log I0507 06:28:20.780473 300 raft_consensus.cc:340] T 00000000000000000000000000000000 P 2f68b13a6f104433814b4b2165a22e0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f68b13a6f104433814b4b2165a22e0e" member_type: VOTER } I0507 06:28:20.780810 300 raft_consensus.cc:366] T 00000000000000000000000000000000 P 2f68b13a6f104433814b4b2165a22e0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0507 06:28:20.780877 300 raft_consensus.cc:705] T 00000000000000000000000000000000 P 2f68b13a6f104433814b4b2165a22e0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f68b13a6f104433814b4b2165a22e0e, State: Initialized, Role: FOLLOWER I0507 06:28:20.781508 300 consensus_queue.cc:229] T 00000000000000000000000000000000 P 2f68b13a6f104433814b4b2165a22e0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f68b13a6f104433814b4b2165a22e0e" member_type: VOTER } I0507 06:28:20.781747 300 raft_consensus.cc:380] T 00000000000000000000000000000000 P 2f68b13a6f104433814b4b2165a22e0e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0507 06:28:20.781834 300 raft_consensus.cc:472] T 00000000000000000000000000000000 P 2f68b13a6f104433814b4b2165a22e0e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0507 06:28:20.781931 300 raft_consensus.cc:2886] T 00000000000000000000000000000000 P 2f68b13a6f104433814b4b2165a22e0e [term 0 FOLLOWER]: Advancing to term 1 I0507 06:28:20.785403 300 raft_consensus.cc:494] T 00000000000000000000000000000000 P 2f68b13a6f104433814b4b2165a22e0e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f68b13a6f104433814b4b2165a22e0e" member_type: VOTER } I0507 06:28:20.785957 300 leader_election.cc:310] T 00000000000000000000000000000000 P 2f68b13a6f104433814b4b2165a22e0e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2f68b13a6f104433814b4b2165a22e0e; no voters: I0507 06:28:20.788750 300 leader_election.cc:296] T 00000000000000000000000000000000 P 2f68b13a6f104433814b4b2165a22e0e [CANDIDATE]: Term 1 election: Requested vote from peers I0507 06:28:20.788954 303 raft_consensus.cc:2640] T 00000000000000000000000000000000 P 2f68b13a6f104433814b4b2165a22e0e [term 1 FOLLOWER]: Leader election won for term 1 I0507 06:28:20.792655 303 raft_consensus.cc:667] T 00000000000000000000000000000000 P 2f68b13a6f104433814b4b2165a22e0e [term 1 LEADER]: Becoming Leader. State: Replica: 2f68b13a6f104433814b4b2165a22e0e, State: Running, Role: LEADER I0507 06:28:20.793082 303 consensus_queue.cc:206] T 00000000000000000000000000000000 P 2f68b13a6f104433814b4b2165a22e0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f68b13a6f104433814b4b2165a22e0e" member_type: VOTER } I0507 06:28:20.801515 304 sys_catalog.cc:337] T 00000000000000000000000000000000 P 2f68b13a6f104433814b4b2165a22e0e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2f68b13a6f104433814b4b2165a22e0e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f68b13a6f104433814b4b2165a22e0e" member_type: VOTER } } I0507 06:28:20.801993 304 sys_catalog.cc:340] T 00000000000000000000000000000000 P 2f68b13a6f104433814b4b2165a22e0e [sys.catalog]: This master's current role is: LEADER I0507 06:28:20.802508 300 sys_catalog.cc:337] T 00000000000000000000000000000000 P 2f68b13a6f104433814b4b2165a22e0e [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 1 leader_uuid: "2f68b13a6f104433814b4b2165a22e0e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f68b13a6f104433814b4b2165a22e0e" member_type: VOTER } } I0507 06:28:20.802513 305 sys_catalog.cc:337] T 00000000000000000000000000000000 P 2f68b13a6f104433814b4b2165a22e0e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2f68b13a6f104433814b4b2165a22e0e. Latest consensus state: current_term: 1 leader_uuid: "2f68b13a6f104433814b4b2165a22e0e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f68b13a6f104433814b4b2165a22e0e" member_type: VOTER } } I0507 06:28:20.802778 300 sys_catalog.cc:340] T 00000000000000000000000000000000 P 2f68b13a6f104433814b4b2165a22e0e [sys.catalog]: This master's current role is: LEADER I0507 06:28:20.802844 305 sys_catalog.cc:340] T 00000000000000000000000000000000 P 2f68b13a6f104433814b4b2165a22e0e [sys.catalog]: This master's current role is: LEADER I0507 06:28:20.804299 300 sys_catalog.cc:424] T 00000000000000000000000000000000 P 2f68b13a6f104433814b4b2165a22e0e [sys.catalog]: configured and running, proceeding with master startup. I0507 06:28:20.804425 309 catalog_manager.cc:1055] Loading table and tablet metadata into memory... I0507 06:28:20.812402 309 catalog_manager.cc:1066] Initializing Kudu internal certificate authority... I0507 06:28:20.875629 309 catalog_manager.cc:950] Generated new certificate authority record I0507 06:28:20.881727 309 catalog_manager.cc:1075] Loading token signing keys... I0507 06:28:20.907114 309 catalog_manager.cc:4292] T 00000000000000000000000000000000 P 2f68b13a6f104433814b4b2165a22e0e: Generated new TSK 0 WARNING: Logging before InitGoogleLogging() is written to STDERR W0507 06:28:20.996248 301 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0507 06:28:20.996553 301 flags.cc:404] Enabled unsafe flag: --never_fsync=true W0507 06:28:21.000689 301 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W0507 06:28:21.000845 301 flags.cc:404] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.57.193 I0507 06:28:21.006489 301 tablet_server_main.cc:78] Tablet server non-default flags: --consensus_rpc_timeout_ms=50 --log_cache_size_limit_mb=10 --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.GetTableAndTabletLocations.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.GetTableAndTabletLocations.1557210500371863-231/raft_consensus-itest-cluster/ts-0/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.0.57.193:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.GetTableAndTabletLocations.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data/info.pb --webserver_interface=127.0.57.193 --webserver_port=0 --tserver_master_addrs=127.0.57.254:42455 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.0.57.193 --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.GetTableAndTabletLocations.1557210500371863-231/raft_consensus-itest-cluster/ts-0/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.10.0-cdh6.x-SNAPSHOT revision 7d0ee26697926e87af9ef395008799577f6ab43f build type FASTDEBUG built by jenkins at 06 May 2019 22:44:21 PST on quasar-knmpfx-1.vpc.cloudera.com build id 322017 ASAN enabled I0507 06:28:21.017035 301 system_ntp.cc:202] NTP initialized. Skew: 500ppm Current error: 25016us I0507 06:28:21.017894 301 fs_manager.cc:263] Metadata directory not provided I0507 06:28:21.017985 301 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0507 06:28:21.018292 301 server_base.cc:433] Could not load existing FS layout: Not found: could not find a healthy instance file I0507 06:28:21.018386 301 server_base.cc:434] Attempting to create new FS layout instead I0507 06:28:21.021724 301 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.GetTableAndTabletLocations.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data/instance: uuid: "d60a9693a9624dafb45798ae3db489de" format_stamp: "Formatted at 2019-05-07 06:28:21 on dist-test-slave-dist-test-slave-rqp6" I0507 06:28:21.022451 301 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.GetTableAndTabletLocations.1557210500371863-231/raft_consensus-itest-cluster/ts-0/wal/instance: uuid: "d60a9693a9624dafb45798ae3db489de" format_stamp: "Formatted at 2019-05-07 06:28:21 on dist-test-slave-dist-test-slave-rqp6" I0507 06:28:21.032130 301 fs_manager.cc:503] Time spent creating directory manager: real 0.009s user 0.002s sys 0.008s I0507 06:28:21.033640 301 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0507 06:28:21.034368 301 file_cache.cc:466] Constructed file cache lbm with capacity 419430 I0507 06:28:21.038153 301 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s I0507 06:28:21.038239 301 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.GetTableAndTabletLocations.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.GetTableAndTabletLocations.1557210500371863-231/raft_consensus-itest-cluster/ts-0/wal uuid: "d60a9693a9624dafb45798ae3db489de" format_stamp: "Formatted at 2019-05-07 06:28:21 on dist-test-slave-dist-test-slave-rqp6" I0507 06:28:21.038482 301 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.GetTableAndTabletLocations.1557210500371863-231/raft_consensus-itest-cluster/ts-0/wal metadata directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.GetTableAndTabletLocations.1557210500371863-231/raft_consensus-itest-cluster/ts-0/wal 1 data directories: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.GetTableAndTabletLocations.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0507 06:28:21.068922 301 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0507 06:28:21.071133 301 ts_tablet_manager.cc:356] Loaded tablet metadata (0 total tablets, 0 live tablets) I0507 06:28:21.071204 301 ts_tablet_manager.cc:376] Registered 0 tablets I0507 06:28:21.114970 301 rpc_server.cc:205] RPC server started. Bound to: 127.0.57.193:33933 I0507 06:28:21.116374 301 webserver.cc:308] Webserver started at http://127.0.57.193:33047/ using document root and password file I0507 06:28:21.117249 301 server_base.cc:608] Dumped server information to /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.GetTableAndTabletLocations.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data/info.pb I0507 06:28:21.123276 231 external_mini_cluster.cc:995] Started /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver as pid 301 I0507 06:28:21.124075 231 external_mini_cluster.cc:933] Running /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.GetTableAndTabletLocations.1557210500371863-231/raft_consensus-itest-cluster/ts-1/wal --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.GetTableAndTabletLocations.1557210500371863-231/raft_consensus-itest-cluster/ts-1/data --block_manager=log --rpc_bind_addresses=127.0.57.194:0 --local_ip_for_outbound_sockets=127.0.57.194 --webserver_interface=127.0.57.194 --webserver_port=0 --tserver_master_addrs=127.0.57.254:42455 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.GetTableAndTabletLocations.1557210500371863-231/raft_consensus-itest-cluster/ts-1/logs --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.GetTableAndTabletLocations.1557210500371863-231/raft_consensus-itest-cluster/ts-1/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --log_cache_size_limit_mb=10 --consensus_rpc_timeout_ms=50 with env {} I0507 06:28:21.146535 431 heartbeater.cc:345] Connected to a master server at 127.0.57.254:42455 I0507 06:28:21.146796 431 heartbeater.cc:426] Registering TS with master... I0507 06:28:21.147442 431 heartbeater.cc:475] Master 127.0.57.254:42455 requested a full tablet report, sending... I0507 06:28:21.149462 267 ts_manager.cc:159] Registered new tserver with Master: d60a9693a9624dafb45798ae3db489de (127.0.57.193:33933) I0507 06:28:21.152870 267 master_service.cc:273] Signed X509 certificate for tserver {username='slave'} at 127.0.57.193:52709 WARNING: Logging before InitGoogleLogging() is written to STDERR W0507 06:28:21.310173 435 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0507 06:28:21.310372 435 flags.cc:404] Enabled unsafe flag: --never_fsync=true W0507 06:28:21.312496 435 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W0507 06:28:21.312642 435 flags.cc:404] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.57.194 I0507 06:28:21.316419 435 tablet_server_main.cc:78] Tablet server non-default flags: --consensus_rpc_timeout_ms=50 --log_cache_size_limit_mb=10 --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.GetTableAndTabletLocations.1557210500371863-231/raft_consensus-itest-cluster/ts-1/data --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.GetTableAndTabletLocations.1557210500371863-231/raft_consensus-itest-cluster/ts-1/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.0.57.194:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.GetTableAndTabletLocations.1557210500371863-231/raft_consensus-itest-cluster/ts-1/data/info.pb --webserver_interface=127.0.57.194 --webserver_port=0 --tserver_master_addrs=127.0.57.254:42455 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.0.57.194 --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.GetTableAndTabletLocations.1557210500371863-231/raft_consensus-itest-cluster/ts-1/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.10.0-cdh6.x-SNAPSHOT revision 7d0ee26697926e87af9ef395008799577f6ab43f build type FASTDEBUG built by jenkins at 06 May 2019 22:44:21 PST on quasar-knmpfx-1.vpc.cloudera.com build id 322017 ASAN enabled I0507 06:28:21.324517 435 system_ntp.cc:202] NTP initialized. Skew: 500ppm Current error: 25016us I0507 06:28:21.325186 435 fs_manager.cc:263] Metadata directory not provided I0507 06:28:21.325248 435 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0507 06:28:21.325475 435 server_base.cc:433] Could not load existing FS layout: Not found: could not find a healthy instance file I0507 06:28:21.325537 435 server_base.cc:434] Attempting to create new FS layout instead I0507 06:28:21.327828 435 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.GetTableAndTabletLocations.1557210500371863-231/raft_consensus-itest-cluster/ts-1/data/instance: uuid: "1641641774fd427e98162710148124a9" format_stamp: "Formatted at 2019-05-07 06:28:21 on dist-test-slave-dist-test-slave-rqp6" I0507 06:28:21.328402 435 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.GetTableAndTabletLocations.1557210500371863-231/raft_consensus-itest-cluster/ts-1/wal/instance: uuid: "1641641774fd427e98162710148124a9" format_stamp: "Formatted at 2019-05-07 06:28:21 on dist-test-slave-dist-test-slave-rqp6" I0507 06:28:21.334462 435 fs_manager.cc:503] Time spent creating directory manager: real 0.006s user 0.002s sys 0.002s I0507 06:28:21.335458 435 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0507 06:28:21.335986 435 file_cache.cc:466] Constructed file cache lbm with capacity 419430 I0507 06:28:21.338850 435 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s I0507 06:28:21.338912 435 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.GetTableAndTabletLocations.1557210500371863-231/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.GetTableAndTabletLocations.1557210500371863-231/raft_consensus-itest-cluster/ts-1/wal uuid: "1641641774fd427e98162710148124a9" format_stamp: "Formatted at 2019-05-07 06:28:21 on dist-test-slave-dist-test-slave-rqp6" I0507 06:28:21.339048 435 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.GetTableAndTabletLocations.1557210500371863-231/raft_consensus-itest-cluster/ts-1/wal metadata directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.GetTableAndTabletLocations.1557210500371863-231/raft_consensus-itest-cluster/ts-1/wal 1 data directories: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.GetTableAndTabletLocations.1557210500371863-231/raft_consensus-itest-cluster/ts-1/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0507 06:28:21.369060 435 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0507 06:28:21.370803 435 ts_tablet_manager.cc:356] Loaded tablet metadata (0 total tablets, 0 live tablets) I0507 06:28:21.370856 435 ts_tablet_manager.cc:376] Registered 0 tablets I0507 06:28:21.399199 435 rpc_server.cc:205] RPC server started. Bound to: 127.0.57.194:43895 I0507 06:28:21.400358 435 webserver.cc:308] Webserver started at http://127.0.57.194:46881/ using document root and password file I0507 06:28:21.401073 435 server_base.cc:608] Dumped server information to /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.GetTableAndTabletLocations.1557210500371863-231/raft_consensus-itest-cluster/ts-1/data/info.pb I0507 06:28:21.410522 231 external_mini_cluster.cc:995] Started /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver as pid 435 I0507 06:28:21.411339 231 external_mini_cluster.cc:933] Running /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.GetTableAndTabletLocations.1557210500371863-231/raft_consensus-itest-cluster/ts-2/wal --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.GetTableAndTabletLocations.1557210500371863-231/raft_consensus-itest-cluster/ts-2/data --block_manager=log --rpc_bind_addresses=127.0.57.195:0 --local_ip_for_outbound_sockets=127.0.57.195 --webserver_interface=127.0.57.195 --webserver_port=0 --tserver_master_addrs=127.0.57.254:42455 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.GetTableAndTabletLocations.1557210500371863-231/raft_consensus-itest-cluster/ts-2/logs --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.GetTableAndTabletLocations.1557210500371863-231/raft_consensus-itest-cluster/ts-2/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --log_cache_size_limit_mb=10 --consensus_rpc_timeout_ms=50 with env {} I0507 06:28:21.416821 554 heartbeater.cc:345] Connected to a master server at 127.0.57.254:42455 I0507 06:28:21.417057 554 heartbeater.cc:426] Registering TS with master... I0507 06:28:21.417696 554 heartbeater.cc:475] Master 127.0.57.254:42455 requested a full tablet report, sending... I0507 06:28:21.418848 267 ts_manager.cc:159] Registered new tserver with Master: 1641641774fd427e98162710148124a9 (127.0.57.194:43895) I0507 06:28:21.419772 267 master_service.cc:273] Signed X509 certificate for tserver {username='slave'} at 127.0.57.194:33709 WARNING: Logging before InitGoogleLogging() is written to STDERR W0507 06:28:21.579409 557 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0507 06:28:21.579643 557 flags.cc:404] Enabled unsafe flag: --never_fsync=true W0507 06:28:21.581766 557 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W0507 06:28:21.581892 557 flags.cc:404] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.57.195 I0507 06:28:21.585672 557 tablet_server_main.cc:78] Tablet server non-default flags: --consensus_rpc_timeout_ms=50 --log_cache_size_limit_mb=10 --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.GetTableAndTabletLocations.1557210500371863-231/raft_consensus-itest-cluster/ts-2/data --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.GetTableAndTabletLocations.1557210500371863-231/raft_consensus-itest-cluster/ts-2/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.0.57.195:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.GetTableAndTabletLocations.1557210500371863-231/raft_consensus-itest-cluster/ts-2/data/info.pb --webserver_interface=127.0.57.195 --webserver_port=0 --tserver_master_addrs=127.0.57.254:42455 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.0.57.195 --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.GetTableAndTabletLocations.1557210500371863-231/raft_consensus-itest-cluster/ts-2/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.10.0-cdh6.x-SNAPSHOT revision 7d0ee26697926e87af9ef395008799577f6ab43f build type FASTDEBUG built by jenkins at 06 May 2019 22:44:21 PST on quasar-knmpfx-1.vpc.cloudera.com build id 322017 ASAN enabled I0507 06:28:21.593176 557 system_ntp.cc:202] NTP initialized. Skew: 500ppm Current error: 25016us I0507 06:28:21.593834 557 fs_manager.cc:263] Metadata directory not provided I0507 06:28:21.593928 557 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0507 06:28:21.594184 557 server_base.cc:433] Could not load existing FS layout: Not found: could not find a healthy instance file I0507 06:28:21.594254 557 server_base.cc:434] Attempting to create new FS layout instead I0507 06:28:21.596659 557 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.GetTableAndTabletLocations.1557210500371863-231/raft_consensus-itest-cluster/ts-2/data/instance: uuid: "498e518111b1402996a07e5bc9bcf0e6" format_stamp: "Formatted at 2019-05-07 06:28:21 on dist-test-slave-dist-test-slave-rqp6" I0507 06:28:21.597949 557 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.GetTableAndTabletLocations.1557210500371863-231/raft_consensus-itest-cluster/ts-2/wal/instance: uuid: "498e518111b1402996a07e5bc9bcf0e6" format_stamp: "Formatted at 2019-05-07 06:28:21 on dist-test-slave-dist-test-slave-rqp6" I0507 06:28:21.604178 557 fs_manager.cc:503] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s I0507 06:28:21.605214 557 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0507 06:28:21.605767 557 file_cache.cc:466] Constructed file cache lbm with capacity 419430 I0507 06:28:21.608816 557 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s I0507 06:28:21.608875 557 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.GetTableAndTabletLocations.1557210500371863-231/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.GetTableAndTabletLocations.1557210500371863-231/raft_consensus-itest-cluster/ts-2/wal uuid: "498e518111b1402996a07e5bc9bcf0e6" format_stamp: "Formatted at 2019-05-07 06:28:21 on dist-test-slave-dist-test-slave-rqp6" I0507 06:28:21.609009 557 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.GetTableAndTabletLocations.1557210500371863-231/raft_consensus-itest-cluster/ts-2/wal metadata directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.GetTableAndTabletLocations.1557210500371863-231/raft_consensus-itest-cluster/ts-2/wal 1 data directories: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.GetTableAndTabletLocations.1557210500371863-231/raft_consensus-itest-cluster/ts-2/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0507 06:28:21.644104 557 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0507 06:28:21.645890 557 ts_tablet_manager.cc:356] Loaded tablet metadata (0 total tablets, 0 live tablets) I0507 06:28:21.645948 557 ts_tablet_manager.cc:376] Registered 0 tablets I0507 06:28:21.673552 557 rpc_server.cc:205] RPC server started. Bound to: 127.0.57.195:42795 I0507 06:28:21.674686 557 webserver.cc:308] Webserver started at http://127.0.57.195:36063/ using document root and password file I0507 06:28:21.675367 557 server_base.cc:608] Dumped server information to /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.GetTableAndTabletLocations.1557210500371863-231/raft_consensus-itest-cluster/ts-2/data/info.pb I0507 06:28:21.685673 231 external_mini_cluster.cc:995] Started /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver as pid 557 I0507 06:28:21.686945 231 external_mini_cluster.cc:933] Running /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.GetTableAndTabletLocations.1557210500371863-231/raft_consensus-itest-cluster/ts-3/wal --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.GetTableAndTabletLocations.1557210500371863-231/raft_consensus-itest-cluster/ts-3/data --block_manager=log --rpc_bind_addresses=127.0.57.196:0 --local_ip_for_outbound_sockets=127.0.57.196 --webserver_interface=127.0.57.196 --webserver_port=0 --tserver_master_addrs=127.0.57.254:42455 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.GetTableAndTabletLocations.1557210500371863-231/raft_consensus-itest-cluster/ts-3/logs --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.GetTableAndTabletLocations.1557210500371863-231/raft_consensus-itest-cluster/ts-3/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --log_cache_size_limit_mb=10 --consensus_rpc_timeout_ms=50 with env {} I0507 06:28:21.688972 676 heartbeater.cc:345] Connected to a master server at 127.0.57.254:42455 I0507 06:28:21.689188 676 heartbeater.cc:426] Registering TS with master... I0507 06:28:21.689739 676 heartbeater.cc:475] Master 127.0.57.254:42455 requested a full tablet report, sending... I0507 06:28:21.690883 267 ts_manager.cc:159] Registered new tserver with Master: 498e518111b1402996a07e5bc9bcf0e6 (127.0.57.195:42795) I0507 06:28:21.691787 267 master_service.cc:273] Signed X509 certificate for tserver {username='slave'} at 127.0.57.195:47761 WARNING: Logging before InitGoogleLogging() is written to STDERR W0507 06:28:21.854905 679 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0507 06:28:21.855108 679 flags.cc:404] Enabled unsafe flag: --never_fsync=true W0507 06:28:21.857149 679 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W0507 06:28:21.857259 679 flags.cc:404] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.57.196 I0507 06:28:21.860939 679 tablet_server_main.cc:78] Tablet server non-default flags: --consensus_rpc_timeout_ms=50 --log_cache_size_limit_mb=10 --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.GetTableAndTabletLocations.1557210500371863-231/raft_consensus-itest-cluster/ts-3/data --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.GetTableAndTabletLocations.1557210500371863-231/raft_consensus-itest-cluster/ts-3/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.0.57.196:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.GetTableAndTabletLocations.1557210500371863-231/raft_consensus-itest-cluster/ts-3/data/info.pb --webserver_interface=127.0.57.196 --webserver_port=0 --tserver_master_addrs=127.0.57.254:42455 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.0.57.196 --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.GetTableAndTabletLocations.1557210500371863-231/raft_consensus-itest-cluster/ts-3/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.10.0-cdh6.x-SNAPSHOT revision 7d0ee26697926e87af9ef395008799577f6ab43f build type FASTDEBUG built by jenkins at 06 May 2019 22:44:21 PST on quasar-knmpfx-1.vpc.cloudera.com build id 322017 ASAN enabled I0507 06:28:21.869060 679 system_ntp.cc:202] NTP initialized. Skew: 500ppm Current error: 25016us I0507 06:28:21.869720 679 fs_manager.cc:263] Metadata directory not provided I0507 06:28:21.869789 679 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0507 06:28:21.870038 679 server_base.cc:433] Could not load existing FS layout: Not found: could not find a healthy instance file I0507 06:28:21.870105 679 server_base.cc:434] Attempting to create new FS layout instead I0507 06:28:21.872745 679 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.GetTableAndTabletLocations.1557210500371863-231/raft_consensus-itest-cluster/ts-3/data/instance: uuid: "39cf9d74193641c49e11b624b77a8a71" format_stamp: "Formatted at 2019-05-07 06:28:21 on dist-test-slave-dist-test-slave-rqp6" I0507 06:28:21.873323 679 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.GetTableAndTabletLocations.1557210500371863-231/raft_consensus-itest-cluster/ts-3/wal/instance: uuid: "39cf9d74193641c49e11b624b77a8a71" format_stamp: "Formatted at 2019-05-07 06:28:21 on dist-test-slave-dist-test-slave-rqp6" I0507 06:28:21.880116 679 fs_manager.cc:503] Time spent creating directory manager: real 0.007s user 0.006s sys 0.000s I0507 06:28:21.881165 679 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0507 06:28:21.881712 679 file_cache.cc:466] Constructed file cache lbm with capacity 419430 I0507 06:28:21.884598 679 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s I0507 06:28:21.884676 679 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.GetTableAndTabletLocations.1557210500371863-231/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.GetTableAndTabletLocations.1557210500371863-231/raft_consensus-itest-cluster/ts-3/wal uuid: "39cf9d74193641c49e11b624b77a8a71" format_stamp: "Formatted at 2019-05-07 06:28:21 on dist-test-slave-dist-test-slave-rqp6" I0507 06:28:21.884843 679 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.GetTableAndTabletLocations.1557210500371863-231/raft_consensus-itest-cluster/ts-3/wal metadata directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.GetTableAndTabletLocations.1557210500371863-231/raft_consensus-itest-cluster/ts-3/wal 1 data directories: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.GetTableAndTabletLocations.1557210500371863-231/raft_consensus-itest-cluster/ts-3/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0507 06:28:21.924576 679 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0507 06:28:21.926250 679 ts_tablet_manager.cc:356] Loaded tablet metadata (0 total tablets, 0 live tablets) I0507 06:28:21.926306 679 ts_tablet_manager.cc:376] Registered 0 tablets I0507 06:28:21.957021 679 rpc_server.cc:205] RPC server started. Bound to: 127.0.57.196:42307 I0507 06:28:21.958123 679 webserver.cc:308] Webserver started at http://127.0.57.196:42079/ using document root and password file I0507 06:28:21.958796 679 server_base.cc:608] Dumped server information to /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.GetTableAndTabletLocations.1557210500371863-231/raft_consensus-itest-cluster/ts-3/data/info.pb I0507 06:28:21.960363 231 external_mini_cluster.cc:995] Started /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver as pid 679 I0507 06:28:21.979159 798 heartbeater.cc:345] Connected to a master server at 127.0.57.254:42455 I0507 06:28:21.979368 798 heartbeater.cc:426] Registering TS with master... I0507 06:28:21.980129 798 heartbeater.cc:475] Master 127.0.57.254:42455 requested a full tablet report, sending... I0507 06:28:21.981294 267 ts_manager.cc:159] Registered new tserver with Master: 39cf9d74193641c49e11b624b77a8a71 (127.0.57.196:42307) I0507 06:28:21.982348 267 master_service.cc:273] Signed X509 certificate for tserver {username='slave'} at 127.0.57.196:39623 I0507 06:28:21.983701 231 external_mini_cluster.cc:623] 4 TS(s) registered with all masters I0507 06:28:22.004199 266 catalog_manager.cc:1415] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33724: name: "TestTable" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 3 split_rows_range_bounds { } partition_schema { range_schema { columns { name: "key" } } } I0507 06:28:22.039059 735 tablet_service.cc:1016] Processing CreateTablet for tablet 140b593c70e7474aaf2ffd0060589754 (table=TestTable [id=1a087e2e5a6f4ffa9afd4eef5924aed8]), partition=RANGE (key) PARTITION UNBOUNDED I0507 06:28:22.039091 613 tablet_service.cc:1016] Processing CreateTablet for tablet 140b593c70e7474aaf2ffd0060589754 (table=TestTable [id=1a087e2e5a6f4ffa9afd4eef5924aed8]), partition=RANGE (key) PARTITION UNBOUNDED I0507 06:28:22.040100 735 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet 140b593c70e7474aaf2ffd0060589754. 1 dirs total, 0 dirs full, 0 dirs failed I0507 06:28:22.040158 613 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet 140b593c70e7474aaf2ffd0060589754. 1 dirs total, 0 dirs full, 0 dirs failed I0507 06:28:22.041373 368 tablet_service.cc:1016] Processing CreateTablet for tablet 140b593c70e7474aaf2ffd0060589754 (table=TestTable [id=1a087e2e5a6f4ffa9afd4eef5924aed8]), partition=RANGE (key) PARTITION UNBOUNDED I0507 06:28:22.042132 368 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet 140b593c70e7474aaf2ffd0060589754. 1 dirs total, 0 dirs full, 0 dirs failed I0507 06:28:22.052955 818 tablet_bootstrap.cc:439] T 140b593c70e7474aaf2ffd0060589754 P d60a9693a9624dafb45798ae3db489de: Bootstrap starting. I0507 06:28:22.055074 818 tablet_bootstrap.cc:590] T 140b593c70e7474aaf2ffd0060589754 P d60a9693a9624dafb45798ae3db489de: No blocks or log segments found. Creating new log. I0507 06:28:22.055691 819 tablet_bootstrap.cc:439] T 140b593c70e7474aaf2ffd0060589754 P 498e518111b1402996a07e5bc9bcf0e6: Bootstrap starting. I0507 06:28:22.057745 818 log.cc:525] T 140b593c70e7474aaf2ffd0060589754 P d60a9693a9624dafb45798ae3db489de: Log is configured to *not* fsync() on all Append() calls I0507 06:28:22.058444 819 tablet_bootstrap.cc:590] T 140b593c70e7474aaf2ffd0060589754 P 498e518111b1402996a07e5bc9bcf0e6: No blocks or log segments found. Creating new log. I0507 06:28:22.059569 820 tablet_bootstrap.cc:439] T 140b593c70e7474aaf2ffd0060589754 P 39cf9d74193641c49e11b624b77a8a71: Bootstrap starting. I0507 06:28:22.060020 819 log.cc:525] T 140b593c70e7474aaf2ffd0060589754 P 498e518111b1402996a07e5bc9bcf0e6: Log is configured to *not* fsync() on all Append() calls I0507 06:28:22.061930 819 tablet_bootstrap.cc:439] T 140b593c70e7474aaf2ffd0060589754 P 498e518111b1402996a07e5bc9bcf0e6: No bootstrap required, opened a new log I0507 06:28:22.062182 819 ts_tablet_manager.cc:1053] T 140b593c70e7474aaf2ffd0060589754 P 498e518111b1402996a07e5bc9bcf0e6: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.003s I0507 06:28:22.062763 820 tablet_bootstrap.cc:590] T 140b593c70e7474aaf2ffd0060589754 P 39cf9d74193641c49e11b624b77a8a71: No blocks or log segments found. Creating new log. I0507 06:28:22.063722 820 log.cc:525] T 140b593c70e7474aaf2ffd0060589754 P 39cf9d74193641c49e11b624b77a8a71: Log is configured to *not* fsync() on all Append() calls I0507 06:28:22.065080 818 tablet_bootstrap.cc:439] T 140b593c70e7474aaf2ffd0060589754 P d60a9693a9624dafb45798ae3db489de: No bootstrap required, opened a new log I0507 06:28:22.065228 820 tablet_bootstrap.cc:439] T 140b593c70e7474aaf2ffd0060589754 P 39cf9d74193641c49e11b624b77a8a71: No bootstrap required, opened a new log I0507 06:28:22.065353 818 ts_tablet_manager.cc:1053] T 140b593c70e7474aaf2ffd0060589754 P d60a9693a9624dafb45798ae3db489de: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.004s I0507 06:28:22.065451 820 ts_tablet_manager.cc:1053] T 140b593c70e7474aaf2ffd0060589754 P 39cf9d74193641c49e11b624b77a8a71: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s I0507 06:28:22.071178 819 raft_consensus.cc:340] T 140b593c70e7474aaf2ffd0060589754 P 498e518111b1402996a07e5bc9bcf0e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39cf9d74193641c49e11b624b77a8a71" member_type: VOTER last_known_addr { host: "127.0.57.196" port: 42307 } } peers { permanent_uuid: "d60a9693a9624dafb45798ae3db489de" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 33933 } } peers { permanent_uuid: "498e518111b1402996a07e5bc9bcf0e6" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 42795 } } I0507 06:28:22.071573 819 raft_consensus.cc:366] T 140b593c70e7474aaf2ffd0060589754 P 498e518111b1402996a07e5bc9bcf0e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0507 06:28:22.071637 819 raft_consensus.cc:705] T 140b593c70e7474aaf2ffd0060589754 P 498e518111b1402996a07e5bc9bcf0e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 498e518111b1402996a07e5bc9bcf0e6, State: Initialized, Role: FOLLOWER I0507 06:28:22.072090 819 consensus_queue.cc:229] T 140b593c70e7474aaf2ffd0060589754 P 498e518111b1402996a07e5bc9bcf0e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39cf9d74193641c49e11b624b77a8a71" member_type: VOTER last_known_addr { host: "127.0.57.196" port: 42307 } } peers { permanent_uuid: "d60a9693a9624dafb45798ae3db489de" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 33933 } } peers { permanent_uuid: "498e518111b1402996a07e5bc9bcf0e6" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 42795 } } I0507 06:28:22.073343 676 heartbeater.cc:467] Master 127.0.57.254:42455 was elected leader, sending a full tablet report... I0507 06:28:22.075207 820 raft_consensus.cc:340] T 140b593c70e7474aaf2ffd0060589754 P 39cf9d74193641c49e11b624b77a8a71 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39cf9d74193641c49e11b624b77a8a71" member_type: VOTER last_known_addr { host: "127.0.57.196" port: 42307 } } peers { permanent_uuid: "d60a9693a9624dafb45798ae3db489de" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 33933 } } peers { permanent_uuid: "498e518111b1402996a07e5bc9bcf0e6" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 42795 } } I0507 06:28:22.075634 820 raft_consensus.cc:366] T 140b593c70e7474aaf2ffd0060589754 P 39cf9d74193641c49e11b624b77a8a71 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0507 06:28:22.075829 820 raft_consensus.cc:705] T 140b593c70e7474aaf2ffd0060589754 P 39cf9d74193641c49e11b624b77a8a71 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39cf9d74193641c49e11b624b77a8a71, State: Initialized, Role: FOLLOWER I0507 06:28:22.076025 819 ts_tablet_manager.cc:1081] T 140b593c70e7474aaf2ffd0060589754 P 498e518111b1402996a07e5bc9bcf0e6: Time spent starting tablet: real 0.014s user 0.006s sys 0.006s I0507 06:28:22.076395 820 consensus_queue.cc:229] T 140b593c70e7474aaf2ffd0060589754 P 39cf9d74193641c49e11b624b77a8a71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39cf9d74193641c49e11b624b77a8a71" member_type: VOTER last_known_addr { host: "127.0.57.196" port: 42307 } } peers { permanent_uuid: "d60a9693a9624dafb45798ae3db489de" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 33933 } } peers { permanent_uuid: "498e518111b1402996a07e5bc9bcf0e6" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 42795 } } I0507 06:28:22.076386 818 raft_consensus.cc:340] T 140b593c70e7474aaf2ffd0060589754 P d60a9693a9624dafb45798ae3db489de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39cf9d74193641c49e11b624b77a8a71" member_type: VOTER last_known_addr { host: "127.0.57.196" port: 42307 } } peers { permanent_uuid: "d60a9693a9624dafb45798ae3db489de" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 33933 } } peers { permanent_uuid: "498e518111b1402996a07e5bc9bcf0e6" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 42795 } } I0507 06:28:22.076855 818 raft_consensus.cc:366] T 140b593c70e7474aaf2ffd0060589754 P d60a9693a9624dafb45798ae3db489de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0507 06:28:22.077428 798 heartbeater.cc:467] Master 127.0.57.254:42455 was elected leader, sending a full tablet report... I0507 06:28:22.077428 818 raft_consensus.cc:705] T 140b593c70e7474aaf2ffd0060589754 P d60a9693a9624dafb45798ae3db489de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d60a9693a9624dafb45798ae3db489de, State: Initialized, Role: FOLLOWER I0507 06:28:22.078068 818 consensus_queue.cc:229] T 140b593c70e7474aaf2ffd0060589754 P d60a9693a9624dafb45798ae3db489de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39cf9d74193641c49e11b624b77a8a71" member_type: VOTER last_known_addr { host: "127.0.57.196" port: 42307 } } peers { permanent_uuid: "d60a9693a9624dafb45798ae3db489de" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 33933 } } peers { permanent_uuid: "498e518111b1402996a07e5bc9bcf0e6" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 42795 } } I0507 06:28:22.079501 820 ts_tablet_manager.cc:1081] T 140b593c70e7474aaf2ffd0060589754 P 39cf9d74193641c49e11b624b77a8a71: Time spent starting tablet: real 0.013s user 0.008s sys 0.004s I0507 06:28:22.080648 431 heartbeater.cc:467] Master 127.0.57.254:42455 was elected leader, sending a full tablet report... I0507 06:28:22.081079 818 ts_tablet_manager.cc:1081] T 140b593c70e7474aaf2ffd0060589754 P d60a9693a9624dafb45798ae3db489de: Time spent starting tablet: real 0.016s user 0.009s sys 0.004s W0507 06:28:22.119969 432 tablet.cc:1786] T 140b593c70e7474aaf2ffd0060589754 P d60a9693a9624dafb45798ae3db489de: Can't schedule compaction. Clean time has not been advanced past its initial value. W0507 06:28:22.177307 677 tablet.cc:1786] T 140b593c70e7474aaf2ffd0060589754 P 498e518111b1402996a07e5bc9bcf0e6: Can't schedule compaction. Clean time has not been advanced past its initial value. W0507 06:28:22.217202 799 tablet.cc:1786] T 140b593c70e7474aaf2ffd0060589754 P 39cf9d74193641c49e11b624b77a8a71: Can't schedule compaction. Clean time has not been advanced past its initial value. I0507 06:28:22.246734 826 raft_consensus.cc:472] T 140b593c70e7474aaf2ffd0060589754 P d60a9693a9624dafb45798ae3db489de [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0507 06:28:22.246949 826 raft_consensus.cc:494] T 140b593c70e7474aaf2ffd0060589754 P d60a9693a9624dafb45798ae3db489de [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39cf9d74193641c49e11b624b77a8a71" member_type: VOTER last_known_addr { host: "127.0.57.196" port: 42307 } } peers { permanent_uuid: "d60a9693a9624dafb45798ae3db489de" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 33933 } } peers { permanent_uuid: "498e518111b1402996a07e5bc9bcf0e6" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 42795 } } I0507 06:28:22.248370 826 leader_election.cc:296] T 140b593c70e7474aaf2ffd0060589754 P d60a9693a9624dafb45798ae3db489de [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 39cf9d74193641c49e11b624b77a8a71 (127.0.57.196:42307), 498e518111b1402996a07e5bc9bcf0e6 (127.0.57.195:42795) I0507 06:28:22.255117 755 tablet_service.cc:1286] Received RequestConsensusVote() RPC: tablet_id: "140b593c70e7474aaf2ffd0060589754" candidate_uuid: "d60a9693a9624dafb45798ae3db489de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39cf9d74193641c49e11b624b77a8a71" is_pre_election: true I0507 06:28:22.255447 755 raft_consensus.cc:2346] T 140b593c70e7474aaf2ffd0060589754 P 39cf9d74193641c49e11b624b77a8a71 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d60a9693a9624dafb45798ae3db489de in term 0. I0507 06:28:22.256070 319 leader_election.cc:310] T 140b593c70e7474aaf2ffd0060589754 P d60a9693a9624dafb45798ae3db489de [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39cf9d74193641c49e11b624b77a8a71, d60a9693a9624dafb45798ae3db489de; no voters: I0507 06:28:22.256155 633 tablet_service.cc:1286] Received RequestConsensusVote() RPC: tablet_id: "140b593c70e7474aaf2ffd0060589754" candidate_uuid: "d60a9693a9624dafb45798ae3db489de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "498e518111b1402996a07e5bc9bcf0e6" is_pre_election: true I0507 06:28:22.256471 633 raft_consensus.cc:2346] T 140b593c70e7474aaf2ffd0060589754 P 498e518111b1402996a07e5bc9bcf0e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d60a9693a9624dafb45798ae3db489de in term 0. I0507 06:28:22.256606 826 raft_consensus.cc:2640] T 140b593c70e7474aaf2ffd0060589754 P d60a9693a9624dafb45798ae3db489de [term 0 FOLLOWER]: Leader pre-election won for term 1 I0507 06:28:22.256700 826 raft_consensus.cc:472] T 140b593c70e7474aaf2ffd0060589754 P d60a9693a9624dafb45798ae3db489de [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0507 06:28:22.256762 826 raft_consensus.cc:2886] T 140b593c70e7474aaf2ffd0060589754 P d60a9693a9624dafb45798ae3db489de [term 0 FOLLOWER]: Advancing to term 1 I0507 06:28:22.259471 826 raft_consensus.cc:494] T 140b593c70e7474aaf2ffd0060589754 P d60a9693a9624dafb45798ae3db489de [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39cf9d74193641c49e11b624b77a8a71" member_type: VOTER last_known_addr { host: "127.0.57.196" port: 42307 } } peers { permanent_uuid: "d60a9693a9624dafb45798ae3db489de" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 33933 } } peers { permanent_uuid: "498e518111b1402996a07e5bc9bcf0e6" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 42795 } } I0507 06:28:22.260046 826 leader_election.cc:296] T 140b593c70e7474aaf2ffd0060589754 P d60a9693a9624dafb45798ae3db489de [CANDIDATE]: Term 1 election: Requested vote from peers 39cf9d74193641c49e11b624b77a8a71 (127.0.57.196:42307), 498e518111b1402996a07e5bc9bcf0e6 (127.0.57.195:42795) I0507 06:28:22.260634 755 tablet_service.cc:1286] Received RequestConsensusVote() RPC: tablet_id: "140b593c70e7474aaf2ffd0060589754" candidate_uuid: "d60a9693a9624dafb45798ae3db489de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39cf9d74193641c49e11b624b77a8a71" I0507 06:28:22.260632 633 tablet_service.cc:1286] Received RequestConsensusVote() RPC: tablet_id: "140b593c70e7474aaf2ffd0060589754" candidate_uuid: "d60a9693a9624dafb45798ae3db489de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "498e518111b1402996a07e5bc9bcf0e6" I0507 06:28:22.260802 755 raft_consensus.cc:2886] T 140b593c70e7474aaf2ffd0060589754 P 39cf9d74193641c49e11b624b77a8a71 [term 0 FOLLOWER]: Advancing to term 1 I0507 06:28:22.260803 633 raft_consensus.cc:2886] T 140b593c70e7474aaf2ffd0060589754 P 498e518111b1402996a07e5bc9bcf0e6 [term 0 FOLLOWER]: Advancing to term 1 I0507 06:28:22.263234 633 raft_consensus.cc:2346] T 140b593c70e7474aaf2ffd0060589754 P 498e518111b1402996a07e5bc9bcf0e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d60a9693a9624dafb45798ae3db489de in term 1. I0507 06:28:22.263234 755 raft_consensus.cc:2346] T 140b593c70e7474aaf2ffd0060589754 P 39cf9d74193641c49e11b624b77a8a71 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d60a9693a9624dafb45798ae3db489de in term 1. I0507 06:28:22.263705 320 leader_election.cc:310] T 140b593c70e7474aaf2ffd0060589754 P d60a9693a9624dafb45798ae3db489de [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 498e518111b1402996a07e5bc9bcf0e6, d60a9693a9624dafb45798ae3db489de; no voters: I0507 06:28:22.264025 826 raft_consensus.cc:2640] T 140b593c70e7474aaf2ffd0060589754 P d60a9693a9624dafb45798ae3db489de [term 1 FOLLOWER]: Leader election won for term 1 I0507 06:28:22.266485 826 raft_consensus.cc:667] T 140b593c70e7474aaf2ffd0060589754 P d60a9693a9624dafb45798ae3db489de [term 1 LEADER]: Becoming Leader. State: Replica: d60a9693a9624dafb45798ae3db489de, State: Running, Role: LEADER I0507 06:28:22.266880 826 consensus_queue.cc:206] T 140b593c70e7474aaf2ffd0060589754 P d60a9693a9624dafb45798ae3db489de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39cf9d74193641c49e11b624b77a8a71" member_type: VOTER last_known_addr { host: "127.0.57.196" port: 42307 } } peers { permanent_uuid: "d60a9693a9624dafb45798ae3db489de" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 33933 } } peers { permanent_uuid: "498e518111b1402996a07e5bc9bcf0e6" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 42795 } } I0507 06:28:22.271566 266 catalog_manager.cc:3988] T 140b593c70e7474aaf2ffd0060589754 P d60a9693a9624dafb45798ae3db489de reported cstate change: term changed from 0 to 1, leader changed from to d60a9693a9624dafb45798ae3db489de (127.0.57.193). New cstate: current_term: 1 leader_uuid: "d60a9693a9624dafb45798ae3db489de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39cf9d74193641c49e11b624b77a8a71" member_type: VOTER last_known_addr { host: "127.0.57.196" port: 42307 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d60a9693a9624dafb45798ae3db489de" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 33933 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "498e518111b1402996a07e5bc9bcf0e6" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 42795 } health_report { overall_health: UNKNOWN } } } I0507 06:28:22.335464 231 external_mini_cluster.cc:623] 4 TS(s) registered with all masters I0507 06:28:22.337172 231 ts_itest-base.cc:232] Waiting for 1 tablets on tserver d60a9693a9624dafb45798ae3db489de to finish bootstrapping I0507 06:28:22.344137 231 ts_itest-base.cc:232] Waiting for 0 tablets on tserver 1641641774fd427e98162710148124a9 to finish bootstrapping I0507 06:28:22.350802 231 ts_itest-base.cc:232] Waiting for 1 tablets on tserver 498e518111b1402996a07e5bc9bcf0e6 to finish bootstrapping I0507 06:28:22.356225 231 ts_itest-base.cc:232] Waiting for 1 tablets on tserver 39cf9d74193641c49e11b624b77a8a71 to finish bootstrapping I0507 06:28:22.422150 554 heartbeater.cc:467] Master 127.0.57.254:42455 was elected leader, sending a full tablet report... I0507 06:28:22.693603 755 raft_consensus.cc:1181] T 140b593c70e7474aaf2ffd0060589754 P 39cf9d74193641c49e11b624b77a8a71 [term 1 FOLLOWER]: Refusing update from remote peer d60a9693a9624dafb45798ae3db489de: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch) I0507 06:28:22.697151 826 consensus_queue.cc:984] T 140b593c70e7474aaf2ffd0060589754 P d60a9693a9624dafb45798ae3db489de [LEADER]: Connected to new peer: Peer: permanent_uuid: "39cf9d74193641c49e11b624b77a8a71" member_type: VOTER last_known_addr { host: "127.0.57.196" port: 42307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0507 06:28:22.703706 633 raft_consensus.cc:1181] T 140b593c70e7474aaf2ffd0060589754 P 498e518111b1402996a07e5bc9bcf0e6 [term 1 FOLLOWER]: Refusing update from remote peer d60a9693a9624dafb45798ae3db489de: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch) I0507 06:28:22.704469 826 consensus_queue.cc:984] T 140b593c70e7474aaf2ffd0060589754 P d60a9693a9624dafb45798ae3db489de [LEADER]: Connected to new peer: Peer: permanent_uuid: "498e518111b1402996a07e5bc9bcf0e6" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 42795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0507 06:28:22.713151 388 consensus_queue.cc:206] T 140b593c70e7474aaf2ffd0060589754 P d60a9693a9624dafb45798ae3db489de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "39cf9d74193641c49e11b624b77a8a71" member_type: VOTER last_known_addr { host: "127.0.57.196" port: 42307 } } peers { permanent_uuid: "d60a9693a9624dafb45798ae3db489de" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 33933 } } peers { permanent_uuid: "498e518111b1402996a07e5bc9bcf0e6" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 42795 } } peers { permanent_uuid: "1641641774fd427e98162710148124a9" member_type: NON_VOTER last_known_addr { host: "127.0.57.194" port: 43895 } attrs { } } I0507 06:28:22.715034 633 raft_consensus.cc:1181] T 140b593c70e7474aaf2ffd0060589754 P 498e518111b1402996a07e5bc9bcf0e6 [term 1 FOLLOWER]: Refusing update from remote peer d60a9693a9624dafb45798ae3db489de: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0507 06:28:22.715220 755 raft_consensus.cc:1181] T 140b593c70e7474aaf2ffd0060589754 P 39cf9d74193641c49e11b624b77a8a71 [term 1 FOLLOWER]: Refusing update from remote peer d60a9693a9624dafb45798ae3db489de: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0507 06:28:22.715509 829 consensus_queue.cc:984] T 140b593c70e7474aaf2ffd0060589754 P d60a9693a9624dafb45798ae3db489de [LEADER]: Connected to new peer: Peer: permanent_uuid: "498e518111b1402996a07e5bc9bcf0e6" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 42795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s I0507 06:28:22.715795 826 consensus_queue.cc:984] T 140b593c70e7474aaf2ffd0060589754 P d60a9693a9624dafb45798ae3db489de [LEADER]: Connected to new peer: Peer: permanent_uuid: "39cf9d74193641c49e11b624b77a8a71" member_type: VOTER last_known_addr { host: "127.0.57.196" port: 42307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s I0507 06:28:22.718102 829 raft_consensus.cc:2788] T 140b593c70e7474aaf2ffd0060589754 P d60a9693a9624dafb45798ae3db489de [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1641641774fd427e98162710148124a9 (127.0.57.194) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "39cf9d74193641c49e11b624b77a8a71" member_type: VOTER last_known_addr { host: "127.0.57.196" port: 42307 } } peers { permanent_uuid: "d60a9693a9624dafb45798ae3db489de" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 33933 } } peers { permanent_uuid: "498e518111b1402996a07e5bc9bcf0e6" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 42795 } } peers { permanent_uuid: "1641641774fd427e98162710148124a9" member_type: NON_VOTER last_known_addr { host: "127.0.57.194" port: 43895 } attrs { } } } I0507 06:28:22.718580 754 raft_consensus.cc:2788] T 140b593c70e7474aaf2ffd0060589754 P 39cf9d74193641c49e11b624b77a8a71 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1641641774fd427e98162710148124a9 (127.0.57.194) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "39cf9d74193641c49e11b624b77a8a71" member_type: VOTER last_known_addr { host: "127.0.57.196" port: 42307 } } peers { permanent_uuid: "d60a9693a9624dafb45798ae3db489de" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 33933 } } peers { permanent_uuid: "498e518111b1402996a07e5bc9bcf0e6" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 42795 } } peers { permanent_uuid: "1641641774fd427e98162710148124a9" member_type: NON_VOTER last_known_addr { host: "127.0.57.194" port: 43895 } attrs { } } } I0507 06:28:22.721139 632 raft_consensus.cc:2788] T 140b593c70e7474aaf2ffd0060589754 P 498e518111b1402996a07e5bc9bcf0e6 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1641641774fd427e98162710148124a9 (127.0.57.194) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "39cf9d74193641c49e11b624b77a8a71" member_type: VOTER last_known_addr { host: "127.0.57.196" port: 42307 } } peers { permanent_uuid: "d60a9693a9624dafb45798ae3db489de" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 33933 } } peers { permanent_uuid: "498e518111b1402996a07e5bc9bcf0e6" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 42795 } } peers { permanent_uuid: "1641641774fd427e98162710148124a9" member_type: NON_VOTER last_known_addr { host: "127.0.57.194" port: 43895 } attrs { } } } I0507 06:28:22.724133 266 catalog_manager.cc:3988] T 140b593c70e7474aaf2ffd0060589754 P 39cf9d74193641c49e11b624b77a8a71 reported cstate change: config changed from index -1 to 2, NON_VOTER 1641641774fd427e98162710148124a9 (127.0.57.194) added. New cstate: current_term: 1 leader_uuid: "d60a9693a9624dafb45798ae3db489de" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "39cf9d74193641c49e11b624b77a8a71" member_type: VOTER last_known_addr { host: "127.0.57.196" port: 42307 } } peers { permanent_uuid: "d60a9693a9624dafb45798ae3db489de" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 33933 } } peers { permanent_uuid: "498e518111b1402996a07e5bc9bcf0e6" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 42795 } } peers { permanent_uuid: "1641641774fd427e98162710148124a9" member_type: NON_VOTER last_known_addr { host: "127.0.57.194" port: 43895 } attrs { } } } W0507 06:28:22.725939 322 consensus_peers.cc:457] T 140b593c70e7474aaf2ffd0060589754 P d60a9693a9624dafb45798ae3db489de -> Peer 1641641774fd427e98162710148124a9 (127.0.57.194:43895): Couldn't send request to peer 1641641774fd427e98162710148124a9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 140b593c70e7474aaf2ffd0060589754. This is attempt 1: this message will repeat every 5th retry. I0507 06:28:23.254742 843 ts_tablet_manager.cc:676] T 140b593c70e7474aaf2ffd0060589754 P 1641641774fd427e98162710148124a9: Initiating tablet copy from peer d60a9693a9624dafb45798ae3db489de (127.0.57.193:33933) I0507 06:28:23.255025 843 tablet_copy_client.cc:240] T 140b593c70e7474aaf2ffd0060589754 P 1641641774fd427e98162710148124a9: tablet copy: Beginning tablet copy session from remote peer at address 127.0.57.193:33933 I0507 06:28:23.260017 408 tablet_copy_service.cc:135] P d60a9693a9624dafb45798ae3db489de: Received BeginTabletCopySession request for tablet 140b593c70e7474aaf2ffd0060589754 from peer 1641641774fd427e98162710148124a9 ({username='slave'} at 127.0.57.194:45127) I0507 06:28:23.260186 408 tablet_copy_service.cc:156] P d60a9693a9624dafb45798ae3db489de: Beginning new tablet copy session on tablet 140b593c70e7474aaf2ffd0060589754 from peer 1641641774fd427e98162710148124a9 at {username='slave'} at 127.0.57.194:45127: session id = 1641641774fd427e98162710148124a9-140b593c70e7474aaf2ffd0060589754 I0507 06:28:23.262531 408 tablet_copy_source_session.cc:202] T 140b593c70e7474aaf2ffd0060589754 P d60a9693a9624dafb45798ae3db489de: Tablet Copy: opened 0 blocks and 1 log segments I0507 06:28:23.265759 843 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet 140b593c70e7474aaf2ffd0060589754. 1 dirs total, 0 dirs full, 0 dirs failed I0507 06:28:23.274834 843 tablet_copy_client.cc:575] T 140b593c70e7474aaf2ffd0060589754 P 1641641774fd427e98162710148124a9: tablet copy: Starting download of 0 data blocks... I0507 06:28:23.275836 843 tablet_copy_client.cc:538] T 140b593c70e7474aaf2ffd0060589754 P 1641641774fd427e98162710148124a9: tablet copy: Starting download of 1 WAL segments... I0507 06:28:23.277559 843 tablet_copy_client.cc:414] T 140b593c70e7474aaf2ffd0060589754 P 1641641774fd427e98162710148124a9: tablet copy: Tablet Copy complete. Replacing tablet superblock. I0507 06:28:23.279912 843 tablet_bootstrap.cc:439] T 140b593c70e7474aaf2ffd0060589754 P 1641641774fd427e98162710148124a9: Bootstrap starting. I0507 06:28:23.285070 843 log.cc:525] T 140b593c70e7474aaf2ffd0060589754 P 1641641774fd427e98162710148124a9: Log is configured to *not* fsync() on all Append() calls I0507 06:28:23.287607 843 tablet_bootstrap.cc:439] T 140b593c70e7474aaf2ffd0060589754 P 1641641774fd427e98162710148124a9: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I0507 06:28:23.287920 843 tablet_bootstrap.cc:439] T 140b593c70e7474aaf2ffd0060589754 P 1641641774fd427e98162710148124a9: Bootstrap complete. I0507 06:28:23.288178 843 ts_tablet_manager.cc:1053] T 140b593c70e7474aaf2ffd0060589754 P 1641641774fd427e98162710148124a9: Time spent bootstrapping tablet: real 0.008s user 0.008s sys 0.000s I0507 06:28:23.294363 843 raft_consensus.cc:340] T 140b593c70e7474aaf2ffd0060589754 P 1641641774fd427e98162710148124a9 [term 1 LEARNER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "39cf9d74193641c49e11b624b77a8a71" member_type: VOTER last_known_addr { host: "127.0.57.196" port: 42307 } } peers { permanent_uuid: "d60a9693a9624dafb45798ae3db489de" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 33933 } } peers { permanent_uuid: "498e518111b1402996a07e5bc9bcf0e6" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 42795 } } peers { permanent_uuid: "1641641774fd427e98162710148124a9" member_type: NON_VOTER last_known_addr { host: "127.0.57.194" port: 43895 } attrs { } } I0507 06:28:23.294706 843 raft_consensus.cc:705] T 140b593c70e7474aaf2ffd0060589754 P 1641641774fd427e98162710148124a9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1641641774fd427e98162710148124a9, State: Initialized, Role: LEARNER I0507 06:28:23.295019 843 consensus_queue.cc:229] T 140b593c70e7474aaf2ffd0060589754 P 1641641774fd427e98162710148124a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "39cf9d74193641c49e11b624b77a8a71" member_type: VOTER last_known_addr { host: "127.0.57.196" port: 42307 } } peers { permanent_uuid: "d60a9693a9624dafb45798ae3db489de" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 33933 } } peers { permanent_uuid: "498e518111b1402996a07e5bc9bcf0e6" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 42795 } } peers { permanent_uuid: "1641641774fd427e98162710148124a9" member_type: NON_VOTER last_known_addr { host: "127.0.57.194" port: 43895 } attrs { } } I0507 06:28:23.296386 843 ts_tablet_manager.cc:1081] T 140b593c70e7474aaf2ffd0060589754 P 1641641774fd427e98162710148124a9: Time spent starting tablet: real 0.008s user 0.001s sys 0.009s I0507 06:28:23.297256 408 tablet_copy_service.cc:325] P d60a9693a9624dafb45798ae3db489de: Request end of tablet copy session 1641641774fd427e98162710148124a9-140b593c70e7474aaf2ffd0060589754 received from {username='slave'} at 127.0.57.194:45127 I0507 06:28:23.297370 408 tablet_copy_service.cc:413] P d60a9693a9624dafb45798ae3db489de: ending tablet copy session 1641641774fd427e98162710148124a9-140b593c70e7474aaf2ffd0060589754 on tablet 140b593c70e7474aaf2ffd0060589754 with peer 1641641774fd427e98162710148124a9 I0507 06:28:23.304262 231 external_mini_cluster.cc:1172] Attempting to check leaks for /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver pid 301 I0507 06:28:23.305172 428 generic_service.cc:171] Checking for leaks (request via RPC) W0507 06:28:23.361735 320 consensus_peers.cc:457] T 140b593c70e7474aaf2ffd0060589754 P d60a9693a9624dafb45798ae3db489de -> Peer 498e518111b1402996a07e5bc9bcf0e6 (127.0.57.195:42795): Couldn't send request to peer 498e518111b1402996a07e5bc9bcf0e6. Status: Timed out: UpdateConsensus RPC to 127.0.57.195:42795 timed out after 0.050s (ON_OUTBOUND_QUEUE). This is attempt 1: this message will repeat every 5th retry. I0507 06:28:23.363112 231 external_mini_cluster.cc:1131] Killing /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver with pid 301 I0507 06:28:23.374255 231 external_mini_cluster.cc:1172] Attempting to check leaks for /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver pid 435 I0507 06:28:23.375181 551 generic_service.cc:171] Checking for leaks (request via RPC) I0507 06:28:23.421093 231 external_mini_cluster.cc:1131] Killing /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver with pid 435 I0507 06:28:23.434134 231 external_mini_cluster.cc:1172] Attempting to check leaks for /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver pid 557 I0507 06:28:23.435115 673 generic_service.cc:171] Checking for leaks (request via RPC) I0507 06:28:23.482353 231 external_mini_cluster.cc:1131] Killing /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver with pid 557 I0507 06:28:23.493510 231 external_mini_cluster.cc:1172] Attempting to check leaks for /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver pid 679 I0507 06:28:23.494515 795 generic_service.cc:171] Checking for leaks (request via RPC) I0507 06:28:23.540755 231 external_mini_cluster.cc:1131] Killing /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver with pid 679 I0507 06:28:23.550725 231 external_mini_cluster.cc:1172] Attempting to check leaks for /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-master pid 241 I0507 06:28:23.551697 297 generic_service.cc:171] Checking for leaks (request via RPC) I0507 06:28:23.626842 231 external_mini_cluster.cc:1131] Killing /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-master with pid 241 [ OK ] RaftConsensusNonVoterITest.GetTableAndTabletLocations (3223 ms) [ RUN ] RaftConsensusNonVoterITest.ReplicaMatchPolicy I0507 06:28:23.649266 231 test_util.cc:212] Using random seed: 1391007709 I0507 06:28:23.650666 231 ts_itest-base.cc:112] Starting cluster with: I0507 06:28:23.650701 231 ts_itest-base.cc:113] -------------- I0507 06:28:23.650732 231 ts_itest-base.cc:114] 4 tablet servers I0507 06:28:23.650759 231 ts_itest-base.cc:115] 3 replicas per TS I0507 06:28:23.650794 231 ts_itest-base.cc:116] -------------- I0507 06:28:23.651943 231 external_mini_cluster.cc:933] Running /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-master /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-master --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.ReplicaMatchPolicy.1557210500371863-231/raft_consensus-itest-cluster/master-0/wal --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.ReplicaMatchPolicy.1557210500371863-231/raft_consensus-itest-cluster/master-0/data --block_manager=log --webserver_interface=localhost --ipki_ca_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.0.57.254:37367 --webserver_interface=127.0.57.254 --webserver_port=0 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.ReplicaMatchPolicy.1557210500371863-231/raft_consensus-itest-cluster/master-0/logs --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.ReplicaMatchPolicy.1557210500371863-231/raft_consensus-itest-cluster/master-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --catalog_manager_evict_excess_replicas=false --rpc_reuseport=true with env {} WARNING: Logging before InitGoogleLogging() is written to STDERR W0507 06:28:23.837245 858 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0507 06:28:23.837507 858 flags.cc:404] Enabled unsafe flag: --never_fsync=true W0507 06:28:23.839915 858 flags.cc:404] Enabled experimental flag: --ipki_ca_key_size=1024 W0507 06:28:23.839967 858 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W0507 06:28:23.840005 858 flags.cc:404] Enabled experimental flag: --tsk_num_rsa_bits=512 W0507 06:28:23.840040 858 flags.cc:404] Enabled experimental flag: --rpc_reuseport=true I0507 06:28:23.844676 858 master_main.cc:73] Master server non-default flags: --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.ReplicaMatchPolicy.1557210500371863-231/raft_consensus-itest-cluster/master-0/data --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.ReplicaMatchPolicy.1557210500371863-231/raft_consensus-itest-cluster/master-0/wal --catalog_manager_evict_excess_replicas=false --ipki_ca_key_size=1024 --ipki_server_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.0.57.254:37367 --rpc_reuseport=true --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.ReplicaMatchPolicy.1557210500371863-231/raft_consensus-itest-cluster/master-0/data/info.pb --webserver_interface=127.0.57.254 --webserver_port=0 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.ReplicaMatchPolicy.1557210500371863-231/raft_consensus-itest-cluster/master-0/logs --logbuflevel=-1 --logtostderr=true Master server version: kudu 1.10.0-cdh6.x-SNAPSHOT revision 7d0ee26697926e87af9ef395008799577f6ab43f build type FASTDEBUG built by jenkins at 06 May 2019 22:44:21 PST on quasar-knmpfx-1.vpc.cloudera.com build id 322017 ASAN enabled I0507 06:28:23.851281 858 system_ntp.cc:202] NTP initialized. Skew: 500ppm Current error: 26016us I0507 06:28:23.852658 858 fs_manager.cc:263] Metadata directory not provided I0507 06:28:23.852764 858 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0507 06:28:23.853111 858 server_base.cc:433] Could not load existing FS layout: Not found: could not find a healthy instance file I0507 06:28:23.853224 858 server_base.cc:434] Attempting to create new FS layout instead I0507 06:28:23.856146 858 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.ReplicaMatchPolicy.1557210500371863-231/raft_consensus-itest-cluster/master-0/data/instance: uuid: "b4205c9647324db1aa1930e75d34b1c7" format_stamp: "Formatted at 2019-05-07 06:28:23 on dist-test-slave-dist-test-slave-rqp6" I0507 06:28:23.856815 858 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.ReplicaMatchPolicy.1557210500371863-231/raft_consensus-itest-cluster/master-0/wal/instance: uuid: "b4205c9647324db1aa1930e75d34b1c7" format_stamp: "Formatted at 2019-05-07 06:28:23 on dist-test-slave-dist-test-slave-rqp6" I0507 06:28:23.863504 858 fs_manager.cc:503] Time spent creating directory manager: real 0.006s user 0.004s sys 0.001s I0507 06:28:23.864660 858 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0507 06:28:23.865232 858 file_cache.cc:466] Constructed file cache lbm with capacity 419430 I0507 06:28:23.868320 858 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s I0507 06:28:23.868474 858 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.ReplicaMatchPolicy.1557210500371863-231/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.ReplicaMatchPolicy.1557210500371863-231/raft_consensus-itest-cluster/master-0/wal uuid: "b4205c9647324db1aa1930e75d34b1c7" format_stamp: "Formatted at 2019-05-07 06:28:23 on dist-test-slave-dist-test-slave-rqp6" I0507 06:28:23.868676 858 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.ReplicaMatchPolicy.1557210500371863-231/raft_consensus-itest-cluster/master-0/wal metadata directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.ReplicaMatchPolicy.1557210500371863-231/raft_consensus-itest-cluster/master-0/wal 1 data directories: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.ReplicaMatchPolicy.1557210500371863-231/raft_consensus-itest-cluster/master-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0507 06:28:23.910656 858 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0507 06:28:23.923947 858 rpc_server.cc:205] RPC server started. Bound to: 127.0.57.254:37367 I0507 06:28:23.925314 858 webserver.cc:308] Webserver started at http://127.0.57.254:37509/ using document root and password file I0507 06:28:23.926144 858 server_base.cc:608] Dumped server information to /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.ReplicaMatchPolicy.1557210500371863-231/raft_consensus-itest-cluster/master-0/data/info.pb I0507 06:28:23.929730 917 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0507 06:28:23.934623 231 external_mini_cluster.cc:995] Started /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-master as pid 858 I0507 06:28:23.935525 231 external_mini_cluster.cc:933] Running /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.ReplicaMatchPolicy.1557210500371863-231/raft_consensus-itest-cluster/ts-0/wal --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.ReplicaMatchPolicy.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data --block_manager=log --rpc_bind_addresses=127.0.57.193:0 --local_ip_for_outbound_sockets=127.0.57.193 --webserver_interface=127.0.57.193 --webserver_port=0 --tserver_master_addrs=127.0.57.254:37367 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.ReplicaMatchPolicy.1557210500371863-231/raft_consensus-itest-cluster/ts-0/logs --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.ReplicaMatchPolicy.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --log_cache_size_limit_mb=10 --consensus_rpc_timeout_ms=50 with env {} I0507 06:28:23.942760 917 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P b4205c9647324db1aa1930e75d34b1c7: Bootstrap starting. I0507 06:28:23.951534 917 tablet_bootstrap.cc:590] T 00000000000000000000000000000000 P b4205c9647324db1aa1930e75d34b1c7: No blocks or log segments found. Creating new log. I0507 06:28:23.956594 917 log.cc:525] T 00000000000000000000000000000000 P b4205c9647324db1aa1930e75d34b1c7: Log is configured to *not* fsync() on all Append() calls I0507 06:28:23.958804 917 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P b4205c9647324db1aa1930e75d34b1c7: No bootstrap required, opened a new log I0507 06:28:23.968264 917 raft_consensus.cc:340] T 00000000000000000000000000000000 P b4205c9647324db1aa1930e75d34b1c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4205c9647324db1aa1930e75d34b1c7" member_type: VOTER } I0507 06:28:23.968624 917 raft_consensus.cc:366] T 00000000000000000000000000000000 P b4205c9647324db1aa1930e75d34b1c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0507 06:28:23.968693 917 raft_consensus.cc:705] T 00000000000000000000000000000000 P b4205c9647324db1aa1930e75d34b1c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4205c9647324db1aa1930e75d34b1c7, State: Initialized, Role: FOLLOWER I0507 06:28:23.969360 917 consensus_queue.cc:229] T 00000000000000000000000000000000 P b4205c9647324db1aa1930e75d34b1c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4205c9647324db1aa1930e75d34b1c7" member_type: VOTER } I0507 06:28:23.969599 917 raft_consensus.cc:380] T 00000000000000000000000000000000 P b4205c9647324db1aa1930e75d34b1c7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0507 06:28:23.969684 917 raft_consensus.cc:472] T 00000000000000000000000000000000 P b4205c9647324db1aa1930e75d34b1c7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0507 06:28:23.969789 917 raft_consensus.cc:2886] T 00000000000000000000000000000000 P b4205c9647324db1aa1930e75d34b1c7 [term 0 FOLLOWER]: Advancing to term 1 I0507 06:28:23.972144 917 raft_consensus.cc:494] T 00000000000000000000000000000000 P b4205c9647324db1aa1930e75d34b1c7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4205c9647324db1aa1930e75d34b1c7" member_type: VOTER } I0507 06:28:23.972723 917 leader_election.cc:310] T 00000000000000000000000000000000 P b4205c9647324db1aa1930e75d34b1c7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b4205c9647324db1aa1930e75d34b1c7; no voters: I0507 06:28:23.976809 917 leader_election.cc:296] T 00000000000000000000000000000000 P b4205c9647324db1aa1930e75d34b1c7 [CANDIDATE]: Term 1 election: Requested vote from peers I0507 06:28:23.977020 920 raft_consensus.cc:2640] T 00000000000000000000000000000000 P b4205c9647324db1aa1930e75d34b1c7 [term 1 FOLLOWER]: Leader election won for term 1 I0507 06:28:23.980720 920 raft_consensus.cc:667] T 00000000000000000000000000000000 P b4205c9647324db1aa1930e75d34b1c7 [term 1 LEADER]: Becoming Leader. State: Replica: b4205c9647324db1aa1930e75d34b1c7, State: Running, Role: LEADER I0507 06:28:23.981567 920 consensus_queue.cc:206] T 00000000000000000000000000000000 P b4205c9647324db1aa1930e75d34b1c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4205c9647324db1aa1930e75d34b1c7" member_type: VOTER } I0507 06:28:23.983819 917 sys_catalog.cc:337] T 00000000000000000000000000000000 P b4205c9647324db1aa1930e75d34b1c7 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 1 leader_uuid: "b4205c9647324db1aa1930e75d34b1c7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4205c9647324db1aa1930e75d34b1c7" member_type: VOTER } } I0507 06:28:23.984279 917 sys_catalog.cc:340] T 00000000000000000000000000000000 P b4205c9647324db1aa1930e75d34b1c7 [sys.catalog]: This master's current role is: LEADER I0507 06:28:23.983922 921 sys_catalog.cc:337] T 00000000000000000000000000000000 P b4205c9647324db1aa1930e75d34b1c7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b4205c9647324db1aa1930e75d34b1c7. Latest consensus state: current_term: 1 leader_uuid: "b4205c9647324db1aa1930e75d34b1c7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4205c9647324db1aa1930e75d34b1c7" member_type: VOTER } } I0507 06:28:23.984302 922 sys_catalog.cc:337] T 00000000000000000000000000000000 P b4205c9647324db1aa1930e75d34b1c7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b4205c9647324db1aa1930e75d34b1c7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4205c9647324db1aa1930e75d34b1c7" member_type: VOTER } } I0507 06:28:23.985260 922 sys_catalog.cc:340] T 00000000000000000000000000000000 P b4205c9647324db1aa1930e75d34b1c7 [sys.catalog]: This master's current role is: LEADER I0507 06:28:23.985201 921 sys_catalog.cc:340] T 00000000000000000000000000000000 P b4205c9647324db1aa1930e75d34b1c7 [sys.catalog]: This master's current role is: LEADER I0507 06:28:23.987037 917 sys_catalog.cc:424] T 00000000000000000000000000000000 P b4205c9647324db1aa1930e75d34b1c7 [sys.catalog]: configured and running, proceeding with master startup. I0507 06:28:23.987200 924 catalog_manager.cc:1055] Loading table and tablet metadata into memory... I0507 06:28:23.995461 924 catalog_manager.cc:1066] Initializing Kudu internal certificate authority... I0507 06:28:24.086668 924 catalog_manager.cc:950] Generated new certificate authority record I0507 06:28:24.088083 924 catalog_manager.cc:1075] Loading token signing keys... I0507 06:28:24.104403 924 catalog_manager.cc:4292] T 00000000000000000000000000000000 P b4205c9647324db1aa1930e75d34b1c7: Generated new TSK 0 WARNING: Logging before InitGoogleLogging() is written to STDERR W0507 06:28:24.143896 918 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0507 06:28:24.144203 918 flags.cc:404] Enabled unsafe flag: --never_fsync=true W0507 06:28:24.147514 918 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W0507 06:28:24.147694 918 flags.cc:404] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.57.193 I0507 06:28:24.160755 918 tablet_server_main.cc:78] Tablet server non-default flags: --consensus_rpc_timeout_ms=50 --log_cache_size_limit_mb=10 --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.ReplicaMatchPolicy.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.ReplicaMatchPolicy.1557210500371863-231/raft_consensus-itest-cluster/ts-0/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.0.57.193:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.ReplicaMatchPolicy.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data/info.pb --webserver_interface=127.0.57.193 --webserver_port=0 --tserver_master_addrs=127.0.57.254:37367 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.0.57.193 --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.ReplicaMatchPolicy.1557210500371863-231/raft_consensus-itest-cluster/ts-0/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.10.0-cdh6.x-SNAPSHOT revision 7d0ee26697926e87af9ef395008799577f6ab43f build type FASTDEBUG built by jenkins at 06 May 2019 22:44:21 PST on quasar-knmpfx-1.vpc.cloudera.com build id 322017 ASAN enabled I0507 06:28:24.171545 918 system_ntp.cc:202] NTP initialized. Skew: 500ppm Current error: 26516us I0507 06:28:24.172425 918 fs_manager.cc:263] Metadata directory not provided I0507 06:28:24.172502 918 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0507 06:28:24.172842 918 server_base.cc:433] Could not load existing FS layout: Not found: could not find a healthy instance file I0507 06:28:24.172942 918 server_base.cc:434] Attempting to create new FS layout instead I0507 06:28:24.176232 918 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.ReplicaMatchPolicy.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data/instance: uuid: "6109a76285154f0aaf369787c892238b" format_stamp: "Formatted at 2019-05-07 06:28:24 on dist-test-slave-dist-test-slave-rqp6" I0507 06:28:24.177017 918 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.ReplicaMatchPolicy.1557210500371863-231/raft_consensus-itest-cluster/ts-0/wal/instance: uuid: "6109a76285154f0aaf369787c892238b" format_stamp: "Formatted at 2019-05-07 06:28:24 on dist-test-slave-dist-test-slave-rqp6" I0507 06:28:24.204509 918 fs_manager.cc:503] Time spent creating directory manager: real 0.027s user 0.004s sys 0.006s I0507 06:28:24.206189 918 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0507 06:28:24.206979 918 file_cache.cc:466] Constructed file cache lbm with capacity 419430 I0507 06:28:24.214664 918 fs_manager.cc:419] Time spent opening block manager: real 0.006s user 0.004s sys 0.000s I0507 06:28:24.214756 918 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.ReplicaMatchPolicy.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.ReplicaMatchPolicy.1557210500371863-231/raft_consensus-itest-cluster/ts-0/wal uuid: "6109a76285154f0aaf369787c892238b" format_stamp: "Formatted at 2019-05-07 06:28:24 on dist-test-slave-dist-test-slave-rqp6" I0507 06:28:24.214970 918 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.ReplicaMatchPolicy.1557210500371863-231/raft_consensus-itest-cluster/ts-0/wal metadata directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.ReplicaMatchPolicy.1557210500371863-231/raft_consensus-itest-cluster/ts-0/wal 1 data directories: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.ReplicaMatchPolicy.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0507 06:28:24.269109 918 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0507 06:28:24.273860 918 ts_tablet_manager.cc:356] Loaded tablet metadata (0 total tablets, 0 live tablets) I0507 06:28:24.273941 918 ts_tablet_manager.cc:376] Registered 0 tablets I0507 06:28:24.406028 918 rpc_server.cc:205] RPC server started. Bound to: 127.0.57.193:46143 I0507 06:28:24.407196 918 webserver.cc:308] Webserver started at http://127.0.57.193:45371/ using document root and password file I0507 06:28:24.407896 918 server_base.cc:608] Dumped server information to /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.ReplicaMatchPolicy.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data/info.pb I0507 06:28:24.417408 231 external_mini_cluster.cc:995] Started /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver as pid 918 I0507 06:28:24.418118 231 external_mini_cluster.cc:933] Running /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.ReplicaMatchPolicy.1557210500371863-231/raft_consensus-itest-cluster/ts-1/wal --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.ReplicaMatchPolicy.1557210500371863-231/raft_consensus-itest-cluster/ts-1/data --block_manager=log --rpc_bind_addresses=127.0.57.194:0 --local_ip_for_outbound_sockets=127.0.57.194 --webserver_interface=127.0.57.194 --webserver_port=0 --tserver_master_addrs=127.0.57.254:37367 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.ReplicaMatchPolicy.1557210500371863-231/raft_consensus-itest-cluster/ts-1/logs --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.ReplicaMatchPolicy.1557210500371863-231/raft_consensus-itest-cluster/ts-1/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --log_cache_size_limit_mb=10 --consensus_rpc_timeout_ms=50 with env {} I0507 06:28:24.453094 1047 heartbeater.cc:345] Connected to a master server at 127.0.57.254:37367 I0507 06:28:24.453289 1047 heartbeater.cc:426] Registering TS with master... I0507 06:28:24.453805 1047 heartbeater.cc:475] Master 127.0.57.254:37367 requested a full tablet report, sending... I0507 06:28:24.455319 884 ts_manager.cc:159] Registered new tserver with Master: 6109a76285154f0aaf369787c892238b (127.0.57.193:46143) I0507 06:28:24.456917 884 master_service.cc:273] Signed X509 certificate for tserver {username='slave'} at 127.0.57.193:40791 WARNING: Logging before InitGoogleLogging() is written to STDERR W0507 06:28:24.583514 1051 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0507 06:28:24.583745 1051 flags.cc:404] Enabled unsafe flag: --never_fsync=true W0507 06:28:24.585819 1051 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W0507 06:28:24.585943 1051 flags.cc:404] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.57.194 I0507 06:28:24.589768 1051 tablet_server_main.cc:78] Tablet server non-default flags: --consensus_rpc_timeout_ms=50 --log_cache_size_limit_mb=10 --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.ReplicaMatchPolicy.1557210500371863-231/raft_consensus-itest-cluster/ts-1/data --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.ReplicaMatchPolicy.1557210500371863-231/raft_consensus-itest-cluster/ts-1/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.0.57.194:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.ReplicaMatchPolicy.1557210500371863-231/raft_consensus-itest-cluster/ts-1/data/info.pb --webserver_interface=127.0.57.194 --webserver_port=0 --tserver_master_addrs=127.0.57.254:37367 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.0.57.194 --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.ReplicaMatchPolicy.1557210500371863-231/raft_consensus-itest-cluster/ts-1/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.10.0-cdh6.x-SNAPSHOT revision 7d0ee26697926e87af9ef395008799577f6ab43f build type FASTDEBUG built by jenkins at 06 May 2019 22:44:21 PST on quasar-knmpfx-1.vpc.cloudera.com build id 322017 ASAN enabled I0507 06:28:24.597777 1051 system_ntp.cc:202] NTP initialized. Skew: 500ppm Current error: 26516us I0507 06:28:24.598431 1051 fs_manager.cc:263] Metadata directory not provided I0507 06:28:24.598508 1051 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0507 06:28:24.598785 1051 server_base.cc:433] Could not load existing FS layout: Not found: could not find a healthy instance file I0507 06:28:24.598871 1051 server_base.cc:434] Attempting to create new FS layout instead I0507 06:28:24.601274 1051 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.ReplicaMatchPolicy.1557210500371863-231/raft_consensus-itest-cluster/ts-1/data/instance: uuid: "84c739a43bcf41b68638d0df1ae8e6ac" format_stamp: "Formatted at 2019-05-07 06:28:24 on dist-test-slave-dist-test-slave-rqp6" I0507 06:28:24.601831 1051 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.ReplicaMatchPolicy.1557210500371863-231/raft_consensus-itest-cluster/ts-1/wal/instance: uuid: "84c739a43bcf41b68638d0df1ae8e6ac" format_stamp: "Formatted at 2019-05-07 06:28:24 on dist-test-slave-dist-test-slave-rqp6" I0507 06:28:24.608283 1051 fs_manager.cc:503] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s I0507 06:28:24.609434 1051 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0507 06:28:24.610013 1051 file_cache.cc:466] Constructed file cache lbm with capacity 419430 I0507 06:28:24.612964 1051 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s I0507 06:28:24.613030 1051 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.ReplicaMatchPolicy.1557210500371863-231/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.ReplicaMatchPolicy.1557210500371863-231/raft_consensus-itest-cluster/ts-1/wal uuid: "84c739a43bcf41b68638d0df1ae8e6ac" format_stamp: "Formatted at 2019-05-07 06:28:24 on dist-test-slave-dist-test-slave-rqp6" I0507 06:28:24.613184 1051 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.ReplicaMatchPolicy.1557210500371863-231/raft_consensus-itest-cluster/ts-1/wal metadata directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.ReplicaMatchPolicy.1557210500371863-231/raft_consensus-itest-cluster/ts-1/wal 1 data directories: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.ReplicaMatchPolicy.1557210500371863-231/raft_consensus-itest-cluster/ts-1/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0507 06:28:24.649307 1051 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0507 06:28:24.651015 1051 ts_tablet_manager.cc:356] Loaded tablet metadata (0 total tablets, 0 live tablets) I0507 06:28:24.651073 1051 ts_tablet_manager.cc:376] Registered 0 tablets I0507 06:28:24.673089 1051 rpc_server.cc:205] RPC server started. Bound to: 127.0.57.194:41441 I0507 06:28:24.674167 1051 webserver.cc:308] Webserver started at http://127.0.57.194:35289/ using document root and password file I0507 06:28:24.674851 1051 server_base.cc:608] Dumped server information to /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.ReplicaMatchPolicy.1557210500371863-231/raft_consensus-itest-cluster/ts-1/data/info.pb I0507 06:28:24.680164 231 external_mini_cluster.cc:995] Started /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver as pid 1051 I0507 06:28:24.680814 231 external_mini_cluster.cc:933] Running /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.ReplicaMatchPolicy.1557210500371863-231/raft_consensus-itest-cluster/ts-2/wal --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.ReplicaMatchPolicy.1557210500371863-231/raft_consensus-itest-cluster/ts-2/data --block_manager=log --rpc_bind_addresses=127.0.57.195:0 --local_ip_for_outbound_sockets=127.0.57.195 --webserver_interface=127.0.57.195 --webserver_port=0 --tserver_master_addrs=127.0.57.254:37367 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.ReplicaMatchPolicy.1557210500371863-231/raft_consensus-itest-cluster/ts-2/logs --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.ReplicaMatchPolicy.1557210500371863-231/raft_consensus-itest-cluster/ts-2/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --log_cache_size_limit_mb=10 --consensus_rpc_timeout_ms=50 with env {} I0507 06:28:24.693034 1170 heartbeater.cc:345] Connected to a master server at 127.0.57.254:37367 I0507 06:28:24.693243 1170 heartbeater.cc:426] Registering TS with master... I0507 06:28:24.693753 1170 heartbeater.cc:475] Master 127.0.57.254:37367 requested a full tablet report, sending... I0507 06:28:24.694937 884 ts_manager.cc:159] Registered new tserver with Master: 84c739a43bcf41b68638d0df1ae8e6ac (127.0.57.194:41441) I0507 06:28:24.696188 884 master_service.cc:273] Signed X509 certificate for tserver {username='slave'} at 127.0.57.194:44483 WARNING: Logging before InitGoogleLogging() is written to STDERR W0507 06:28:24.834512 1173 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0507 06:28:24.834758 1173 flags.cc:404] Enabled unsafe flag: --never_fsync=true W0507 06:28:24.836881 1173 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W0507 06:28:24.836982 1173 flags.cc:404] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.57.195 I0507 06:28:24.840721 1173 tablet_server_main.cc:78] Tablet server non-default flags: --consensus_rpc_timeout_ms=50 --log_cache_size_limit_mb=10 --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.ReplicaMatchPolicy.1557210500371863-231/raft_consensus-itest-cluster/ts-2/data --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.ReplicaMatchPolicy.1557210500371863-231/raft_consensus-itest-cluster/ts-2/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.0.57.195:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.ReplicaMatchPolicy.1557210500371863-231/raft_consensus-itest-cluster/ts-2/data/info.pb --webserver_interface=127.0.57.195 --webserver_port=0 --tserver_master_addrs=127.0.57.254:37367 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.0.57.195 --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.ReplicaMatchPolicy.1557210500371863-231/raft_consensus-itest-cluster/ts-2/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.10.0-cdh6.x-SNAPSHOT revision 7d0ee26697926e87af9ef395008799577f6ab43f build type FASTDEBUG built by jenkins at 06 May 2019 22:44:21 PST on quasar-knmpfx-1.vpc.cloudera.com build id 322017 ASAN enabled I0507 06:28:24.848273 1173 system_ntp.cc:202] NTP initialized. Skew: 500ppm Current error: 26516us I0507 06:28:24.848978 1173 fs_manager.cc:263] Metadata directory not provided I0507 06:28:24.849048 1173 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0507 06:28:24.849326 1173 server_base.cc:433] Could not load existing FS layout: Not found: could not find a healthy instance file I0507 06:28:24.849403 1173 server_base.cc:434] Attempting to create new FS layout instead I0507 06:28:24.852087 1173 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.ReplicaMatchPolicy.1557210500371863-231/raft_consensus-itest-cluster/ts-2/data/instance: uuid: "bc45a661e9494fc0bd570e9c5f333f05" format_stamp: "Formatted at 2019-05-07 06:28:24 on dist-test-slave-dist-test-slave-rqp6" I0507 06:28:24.852789 1173 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.ReplicaMatchPolicy.1557210500371863-231/raft_consensus-itest-cluster/ts-2/wal/instance: uuid: "bc45a661e9494fc0bd570e9c5f333f05" format_stamp: "Formatted at 2019-05-07 06:28:24 on dist-test-slave-dist-test-slave-rqp6" I0507 06:28:24.860086 1173 fs_manager.cc:503] Time spent creating directory manager: real 0.007s user 0.006s sys 0.000s I0507 06:28:24.861209 1173 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0507 06:28:24.861773 1173 file_cache.cc:466] Constructed file cache lbm with capacity 419430 I0507 06:28:24.864781 1173 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s I0507 06:28:24.864842 1173 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.ReplicaMatchPolicy.1557210500371863-231/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.ReplicaMatchPolicy.1557210500371863-231/raft_consensus-itest-cluster/ts-2/wal uuid: "bc45a661e9494fc0bd570e9c5f333f05" format_stamp: "Formatted at 2019-05-07 06:28:24 on dist-test-slave-dist-test-slave-rqp6" I0507 06:28:24.865015 1173 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.ReplicaMatchPolicy.1557210500371863-231/raft_consensus-itest-cluster/ts-2/wal metadata directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.ReplicaMatchPolicy.1557210500371863-231/raft_consensus-itest-cluster/ts-2/wal 1 data directories: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.ReplicaMatchPolicy.1557210500371863-231/raft_consensus-itest-cluster/ts-2/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0507 06:28:24.901808 1173 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0507 06:28:24.903458 1173 ts_tablet_manager.cc:356] Loaded tablet metadata (0 total tablets, 0 live tablets) I0507 06:28:24.903515 1173 ts_tablet_manager.cc:376] Registered 0 tablets I0507 06:28:24.936589 1173 rpc_server.cc:205] RPC server started. Bound to: 127.0.57.195:45153 I0507 06:28:24.937662 1173 webserver.cc:308] Webserver started at http://127.0.57.195:39903/ using document root and password file I0507 06:28:24.938318 1173 server_base.cc:608] Dumped server information to /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.ReplicaMatchPolicy.1557210500371863-231/raft_consensus-itest-cluster/ts-2/data/info.pb I0507 06:28:24.945909 231 external_mini_cluster.cc:995] Started /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver as pid 1173 I0507 06:28:24.946637 231 external_mini_cluster.cc:933] Running /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.ReplicaMatchPolicy.1557210500371863-231/raft_consensus-itest-cluster/ts-3/wal --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.ReplicaMatchPolicy.1557210500371863-231/raft_consensus-itest-cluster/ts-3/data --block_manager=log --rpc_bind_addresses=127.0.57.196:0 --local_ip_for_outbound_sockets=127.0.57.196 --webserver_interface=127.0.57.196 --webserver_port=0 --tserver_master_addrs=127.0.57.254:37367 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.ReplicaMatchPolicy.1557210500371863-231/raft_consensus-itest-cluster/ts-3/logs --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.ReplicaMatchPolicy.1557210500371863-231/raft_consensus-itest-cluster/ts-3/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --log_cache_size_limit_mb=10 --consensus_rpc_timeout_ms=50 with env {} I0507 06:28:24.956374 1292 heartbeater.cc:345] Connected to a master server at 127.0.57.254:37367 I0507 06:28:24.956605 1292 heartbeater.cc:426] Registering TS with master... I0507 06:28:24.957123 1292 heartbeater.cc:475] Master 127.0.57.254:37367 requested a full tablet report, sending... I0507 06:28:24.958329 884 ts_manager.cc:159] Registered new tserver with Master: bc45a661e9494fc0bd570e9c5f333f05 (127.0.57.195:45153) I0507 06:28:24.959704 884 master_service.cc:273] Signed X509 certificate for tserver {username='slave'} at 127.0.57.195:45561 WARNING: Logging before InitGoogleLogging() is written to STDERR W0507 06:28:25.098806 1295 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0507 06:28:25.099023 1295 flags.cc:404] Enabled unsafe flag: --never_fsync=true W0507 06:28:25.101263 1295 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W0507 06:28:25.101388 1295 flags.cc:404] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.57.196 I0507 06:28:25.105669 1295 tablet_server_main.cc:78] Tablet server non-default flags: --consensus_rpc_timeout_ms=50 --log_cache_size_limit_mb=10 --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.ReplicaMatchPolicy.1557210500371863-231/raft_consensus-itest-cluster/ts-3/data --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.ReplicaMatchPolicy.1557210500371863-231/raft_consensus-itest-cluster/ts-3/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.0.57.196:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.ReplicaMatchPolicy.1557210500371863-231/raft_consensus-itest-cluster/ts-3/data/info.pb --webserver_interface=127.0.57.196 --webserver_port=0 --tserver_master_addrs=127.0.57.254:37367 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.0.57.196 --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.ReplicaMatchPolicy.1557210500371863-231/raft_consensus-itest-cluster/ts-3/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.10.0-cdh6.x-SNAPSHOT revision 7d0ee26697926e87af9ef395008799577f6ab43f build type FASTDEBUG built by jenkins at 06 May 2019 22:44:21 PST on quasar-knmpfx-1.vpc.cloudera.com build id 322017 ASAN enabled I0507 06:28:25.113173 1295 system_ntp.cc:202] NTP initialized. Skew: 500ppm Current error: 27016us I0507 06:28:25.113822 1295 fs_manager.cc:263] Metadata directory not provided I0507 06:28:25.113893 1295 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0507 06:28:25.114130 1295 server_base.cc:433] Could not load existing FS layout: Not found: could not find a healthy instance file I0507 06:28:25.114194 1295 server_base.cc:434] Attempting to create new FS layout instead I0507 06:28:25.116618 1295 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.ReplicaMatchPolicy.1557210500371863-231/raft_consensus-itest-cluster/ts-3/data/instance: uuid: "1df6c947010648f2aa8c01c2f5c07ed7" format_stamp: "Formatted at 2019-05-07 06:28:25 on dist-test-slave-dist-test-slave-rqp6" I0507 06:28:25.117153 1295 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.ReplicaMatchPolicy.1557210500371863-231/raft_consensus-itest-cluster/ts-3/wal/instance: uuid: "1df6c947010648f2aa8c01c2f5c07ed7" format_stamp: "Formatted at 2019-05-07 06:28:25 on dist-test-slave-dist-test-slave-rqp6" I0507 06:28:25.123641 1295 fs_manager.cc:503] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s I0507 06:28:25.124722 1295 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0507 06:28:25.125306 1295 file_cache.cc:466] Constructed file cache lbm with capacity 419430 I0507 06:28:25.128162 1295 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s I0507 06:28:25.128221 1295 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.ReplicaMatchPolicy.1557210500371863-231/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.ReplicaMatchPolicy.1557210500371863-231/raft_consensus-itest-cluster/ts-3/wal uuid: "1df6c947010648f2aa8c01c2f5c07ed7" format_stamp: "Formatted at 2019-05-07 06:28:25 on dist-test-slave-dist-test-slave-rqp6" I0507 06:28:25.128423 1295 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.ReplicaMatchPolicy.1557210500371863-231/raft_consensus-itest-cluster/ts-3/wal metadata directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.ReplicaMatchPolicy.1557210500371863-231/raft_consensus-itest-cluster/ts-3/wal 1 data directories: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.ReplicaMatchPolicy.1557210500371863-231/raft_consensus-itest-cluster/ts-3/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0507 06:28:25.156328 1295 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0507 06:28:25.158025 1295 ts_tablet_manager.cc:356] Loaded tablet metadata (0 total tablets, 0 live tablets) I0507 06:28:25.158084 1295 ts_tablet_manager.cc:376] Registered 0 tablets I0507 06:28:25.180668 1295 rpc_server.cc:205] RPC server started. Bound to: 127.0.57.196:41459 I0507 06:28:25.181845 1295 webserver.cc:308] Webserver started at http://127.0.57.196:33963/ using document root and password file I0507 06:28:25.182513 1295 server_base.cc:608] Dumped server information to /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.ReplicaMatchPolicy.1557210500371863-231/raft_consensus-itest-cluster/ts-3/data/info.pb I0507 06:28:25.189608 231 external_mini_cluster.cc:995] Started /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver as pid 1295 I0507 06:28:25.195789 1414 heartbeater.cc:345] Connected to a master server at 127.0.57.254:37367 I0507 06:28:25.196010 1414 heartbeater.cc:426] Registering TS with master... I0507 06:28:25.196676 1414 heartbeater.cc:475] Master 127.0.57.254:37367 requested a full tablet report, sending... I0507 06:28:25.197847 884 ts_manager.cc:159] Registered new tserver with Master: 1df6c947010648f2aa8c01c2f5c07ed7 (127.0.57.196:41459) I0507 06:28:25.198812 884 master_service.cc:273] Signed X509 certificate for tserver {username='slave'} at 127.0.57.196:57127 I0507 06:28:25.199434 231 external_mini_cluster.cc:623] 4 TS(s) registered with all masters I0507 06:28:25.210464 884 catalog_manager.cc:1415] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36600: name: "TestTable" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 3 split_rows_range_bounds { } partition_schema { range_schema { columns { name: "key" } } } I0507 06:28:25.244037 1229 tablet_service.cc:1016] Processing CreateTablet for tablet 7d491c16d4ec46119cccd7f4c91f3df1 (table=TestTable [id=07a2d0f6d3fa4f36b5cda35f6c677061]), partition=RANGE (key) PARTITION UNBOUNDED I0507 06:28:25.245059 1229 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet 7d491c16d4ec46119cccd7f4c91f3df1. 1 dirs total, 0 dirs full, 0 dirs failed I0507 06:28:25.245185 1351 tablet_service.cc:1016] Processing CreateTablet for tablet 7d491c16d4ec46119cccd7f4c91f3df1 (table=TestTable [id=07a2d0f6d3fa4f36b5cda35f6c677061]), partition=RANGE (key) PARTITION UNBOUNDED I0507 06:28:25.245184 1107 tablet_service.cc:1016] Processing CreateTablet for tablet 7d491c16d4ec46119cccd7f4c91f3df1 (table=TestTable [id=07a2d0f6d3fa4f36b5cda35f6c677061]), partition=RANGE (key) PARTITION UNBOUNDED I0507 06:28:25.246091 1107 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet 7d491c16d4ec46119cccd7f4c91f3df1. 1 dirs total, 0 dirs full, 0 dirs failed I0507 06:28:25.246095 1351 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet 7d491c16d4ec46119cccd7f4c91f3df1. 1 dirs total, 0 dirs full, 0 dirs failed I0507 06:28:25.255156 1434 tablet_bootstrap.cc:439] T 7d491c16d4ec46119cccd7f4c91f3df1 P bc45a661e9494fc0bd570e9c5f333f05: Bootstrap starting. I0507 06:28:25.257447 1434 tablet_bootstrap.cc:590] T 7d491c16d4ec46119cccd7f4c91f3df1 P bc45a661e9494fc0bd570e9c5f333f05: No blocks or log segments found. Creating new log. I0507 06:28:25.258828 1435 tablet_bootstrap.cc:439] T 7d491c16d4ec46119cccd7f4c91f3df1 P 84c739a43bcf41b68638d0df1ae8e6ac: Bootstrap starting. I0507 06:28:25.259816 1436 tablet_bootstrap.cc:439] T 7d491c16d4ec46119cccd7f4c91f3df1 P 1df6c947010648f2aa8c01c2f5c07ed7: Bootstrap starting. I0507 06:28:25.262598 1435 tablet_bootstrap.cc:590] T 7d491c16d4ec46119cccd7f4c91f3df1 P 84c739a43bcf41b68638d0df1ae8e6ac: No blocks or log segments found. Creating new log. I0507 06:28:25.263494 1435 log.cc:525] T 7d491c16d4ec46119cccd7f4c91f3df1 P 84c739a43bcf41b68638d0df1ae8e6ac: Log is configured to *not* fsync() on all Append() calls I0507 06:28:25.265287 1436 tablet_bootstrap.cc:590] T 7d491c16d4ec46119cccd7f4c91f3df1 P 1df6c947010648f2aa8c01c2f5c07ed7: No blocks or log segments found. Creating new log. I0507 06:28:25.266142 1436 log.cc:525] T 7d491c16d4ec46119cccd7f4c91f3df1 P 1df6c947010648f2aa8c01c2f5c07ed7: Log is configured to *not* fsync() on all Append() calls I0507 06:28:25.266942 1434 log.cc:525] T 7d491c16d4ec46119cccd7f4c91f3df1 P bc45a661e9494fc0bd570e9c5f333f05: Log is configured to *not* fsync() on all Append() calls I0507 06:28:25.268561 1436 tablet_bootstrap.cc:439] T 7d491c16d4ec46119cccd7f4c91f3df1 P 1df6c947010648f2aa8c01c2f5c07ed7: No bootstrap required, opened a new log I0507 06:28:25.268816 1436 ts_tablet_manager.cc:1053] T 7d491c16d4ec46119cccd7f4c91f3df1 P 1df6c947010648f2aa8c01c2f5c07ed7: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.002s I0507 06:28:25.268862 1435 tablet_bootstrap.cc:439] T 7d491c16d4ec46119cccd7f4c91f3df1 P 84c739a43bcf41b68638d0df1ae8e6ac: No bootstrap required, opened a new log I0507 06:28:25.268919 1434 tablet_bootstrap.cc:439] T 7d491c16d4ec46119cccd7f4c91f3df1 P bc45a661e9494fc0bd570e9c5f333f05: No bootstrap required, opened a new log I0507 06:28:25.269129 1434 ts_tablet_manager.cc:1053] T 7d491c16d4ec46119cccd7f4c91f3df1 P bc45a661e9494fc0bd570e9c5f333f05: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.000s I0507 06:28:25.269134 1435 ts_tablet_manager.cc:1053] T 7d491c16d4ec46119cccd7f4c91f3df1 P 84c739a43bcf41b68638d0df1ae8e6ac: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.000s I0507 06:28:25.274914 1436 raft_consensus.cc:340] T 7d491c16d4ec46119cccd7f4c91f3df1 P 1df6c947010648f2aa8c01c2f5c07ed7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc45a661e9494fc0bd570e9c5f333f05" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 45153 } } peers { permanent_uuid: "84c739a43bcf41b68638d0df1ae8e6ac" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 41441 } } peers { permanent_uuid: "1df6c947010648f2aa8c01c2f5c07ed7" member_type: VOTER last_known_addr { host: "127.0.57.196" port: 41459 } } I0507 06:28:25.275324 1436 raft_consensus.cc:366] T 7d491c16d4ec46119cccd7f4c91f3df1 P 1df6c947010648f2aa8c01c2f5c07ed7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0507 06:28:25.275388 1436 raft_consensus.cc:705] T 7d491c16d4ec46119cccd7f4c91f3df1 P 1df6c947010648f2aa8c01c2f5c07ed7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1df6c947010648f2aa8c01c2f5c07ed7, State: Initialized, Role: FOLLOWER I0507 06:28:25.275874 1436 consensus_queue.cc:229] T 7d491c16d4ec46119cccd7f4c91f3df1 P 1df6c947010648f2aa8c01c2f5c07ed7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc45a661e9494fc0bd570e9c5f333f05" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 45153 } } peers { permanent_uuid: "84c739a43bcf41b68638d0df1ae8e6ac" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 41441 } } peers { permanent_uuid: "1df6c947010648f2aa8c01c2f5c07ed7" member_type: VOTER last_known_addr { host: "127.0.57.196" port: 41459 } } I0507 06:28:25.277068 1414 heartbeater.cc:467] Master 127.0.57.254:37367 was elected leader, sending a full tablet report... I0507 06:28:25.277101 1435 raft_consensus.cc:340] T 7d491c16d4ec46119cccd7f4c91f3df1 P 84c739a43bcf41b68638d0df1ae8e6ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc45a661e9494fc0bd570e9c5f333f05" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 45153 } } peers { permanent_uuid: "84c739a43bcf41b68638d0df1ae8e6ac" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 41441 } } peers { permanent_uuid: "1df6c947010648f2aa8c01c2f5c07ed7" member_type: VOTER last_known_addr { host: "127.0.57.196" port: 41459 } } I0507 06:28:25.277525 1435 raft_consensus.cc:366] T 7d491c16d4ec46119cccd7f4c91f3df1 P 84c739a43bcf41b68638d0df1ae8e6ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0507 06:28:25.277581 1435 raft_consensus.cc:705] T 7d491c16d4ec46119cccd7f4c91f3df1 P 84c739a43bcf41b68638d0df1ae8e6ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84c739a43bcf41b68638d0df1ae8e6ac, State: Initialized, Role: FOLLOWER I0507 06:28:25.277598 1436 ts_tablet_manager.cc:1081] T 7d491c16d4ec46119cccd7f4c91f3df1 P 1df6c947010648f2aa8c01c2f5c07ed7: Time spent starting tablet: real 0.009s user 0.007s sys 0.001s I0507 06:28:25.277360 1434 raft_consensus.cc:340] T 7d491c16d4ec46119cccd7f4c91f3df1 P bc45a661e9494fc0bd570e9c5f333f05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc45a661e9494fc0bd570e9c5f333f05" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 45153 } } peers { permanent_uuid: "84c739a43bcf41b68638d0df1ae8e6ac" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 41441 } } peers { permanent_uuid: "1df6c947010648f2aa8c01c2f5c07ed7" member_type: VOTER last_known_addr { host: "127.0.57.196" port: 41459 } } I0507 06:28:25.277726 1434 raft_consensus.cc:366] T 7d491c16d4ec46119cccd7f4c91f3df1 P bc45a661e9494fc0bd570e9c5f333f05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0507 06:28:25.277774 1434 raft_consensus.cc:705] T 7d491c16d4ec46119cccd7f4c91f3df1 P bc45a661e9494fc0bd570e9c5f333f05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc45a661e9494fc0bd570e9c5f333f05, State: Initialized, Role: FOLLOWER I0507 06:28:25.278158 1434 consensus_queue.cc:229] T 7d491c16d4ec46119cccd7f4c91f3df1 P bc45a661e9494fc0bd570e9c5f333f05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc45a661e9494fc0bd570e9c5f333f05" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 45153 } } peers { permanent_uuid: "84c739a43bcf41b68638d0df1ae8e6ac" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 41441 } } peers { permanent_uuid: "1df6c947010648f2aa8c01c2f5c07ed7" member_type: VOTER last_known_addr { host: "127.0.57.196" port: 41459 } } I0507 06:28:25.278142 1435 consensus_queue.cc:229] T 7d491c16d4ec46119cccd7f4c91f3df1 P 84c739a43bcf41b68638d0df1ae8e6ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc45a661e9494fc0bd570e9c5f333f05" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 45153 } } peers { permanent_uuid: "84c739a43bcf41b68638d0df1ae8e6ac" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 41441 } } peers { permanent_uuid: "1df6c947010648f2aa8c01c2f5c07ed7" member_type: VOTER last_known_addr { host: "127.0.57.196" port: 41459 } } I0507 06:28:25.279100 1292 heartbeater.cc:467] Master 127.0.57.254:37367 was elected leader, sending a full tablet report... I0507 06:28:25.279386 1170 heartbeater.cc:467] Master 127.0.57.254:37367 was elected leader, sending a full tablet report... I0507 06:28:25.279985 1434 ts_tablet_manager.cc:1081] T 7d491c16d4ec46119cccd7f4c91f3df1 P bc45a661e9494fc0bd570e9c5f333f05: Time spent starting tablet: real 0.011s user 0.010s sys 0.000s I0507 06:28:25.280709 1435 ts_tablet_manager.cc:1081] T 7d491c16d4ec46119cccd7f4c91f3df1 P 84c739a43bcf41b68638d0df1ae8e6ac: Time spent starting tablet: real 0.012s user 0.007s sys 0.005s I0507 06:28:25.289361 1442 raft_consensus.cc:472] T 7d491c16d4ec46119cccd7f4c91f3df1 P 84c739a43bcf41b68638d0df1ae8e6ac [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0507 06:28:25.289554 1442 raft_consensus.cc:494] T 7d491c16d4ec46119cccd7f4c91f3df1 P 84c739a43bcf41b68638d0df1ae8e6ac [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc45a661e9494fc0bd570e9c5f333f05" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 45153 } } peers { permanent_uuid: "84c739a43bcf41b68638d0df1ae8e6ac" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 41441 } } peers { permanent_uuid: "1df6c947010648f2aa8c01c2f5c07ed7" member_type: VOTER last_known_addr { host: "127.0.57.196" port: 41459 } } I0507 06:28:25.290721 1442 leader_election.cc:296] T 7d491c16d4ec46119cccd7f4c91f3df1 P 84c739a43bcf41b68638d0df1ae8e6ac [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bc45a661e9494fc0bd570e9c5f333f05 (127.0.57.195:45153), 1df6c947010648f2aa8c01c2f5c07ed7 (127.0.57.196:41459) I0507 06:28:25.301595 1249 tablet_service.cc:1286] Received RequestConsensusVote() RPC: tablet_id: "7d491c16d4ec46119cccd7f4c91f3df1" candidate_uuid: "84c739a43bcf41b68638d0df1ae8e6ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc45a661e9494fc0bd570e9c5f333f05" is_pre_election: true I0507 06:28:25.301949 1249 raft_consensus.cc:2346] T 7d491c16d4ec46119cccd7f4c91f3df1 P bc45a661e9494fc0bd570e9c5f333f05 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84c739a43bcf41b68638d0df1ae8e6ac in term 0. I0507 06:28:25.301962 1371 tablet_service.cc:1286] Received RequestConsensusVote() RPC: tablet_id: "7d491c16d4ec46119cccd7f4c91f3df1" candidate_uuid: "84c739a43bcf41b68638d0df1ae8e6ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1df6c947010648f2aa8c01c2f5c07ed7" is_pre_election: true I0507 06:28:25.302312 1371 raft_consensus.cc:2346] T 7d491c16d4ec46119cccd7f4c91f3df1 P 1df6c947010648f2aa8c01c2f5c07ed7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84c739a43bcf41b68638d0df1ae8e6ac in term 0. I0507 06:28:25.302579 1059 leader_election.cc:310] T 7d491c16d4ec46119cccd7f4c91f3df1 P 84c739a43bcf41b68638d0df1ae8e6ac [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 84c739a43bcf41b68638d0df1ae8e6ac, bc45a661e9494fc0bd570e9c5f333f05; no voters: I0507 06:28:25.303004 1442 raft_consensus.cc:2640] T 7d491c16d4ec46119cccd7f4c91f3df1 P 84c739a43bcf41b68638d0df1ae8e6ac [term 0 FOLLOWER]: Leader pre-election won for term 1 I0507 06:28:25.303099 1442 raft_consensus.cc:472] T 7d491c16d4ec46119cccd7f4c91f3df1 P 84c739a43bcf41b68638d0df1ae8e6ac [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0507 06:28:25.303153 1442 raft_consensus.cc:2886] T 7d491c16d4ec46119cccd7f4c91f3df1 P 84c739a43bcf41b68638d0df1ae8e6ac [term 0 FOLLOWER]: Advancing to term 1 I0507 06:28:25.304919 1442 raft_consensus.cc:494] T 7d491c16d4ec46119cccd7f4c91f3df1 P 84c739a43bcf41b68638d0df1ae8e6ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc45a661e9494fc0bd570e9c5f333f05" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 45153 } } peers { permanent_uuid: "84c739a43bcf41b68638d0df1ae8e6ac" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 41441 } } peers { permanent_uuid: "1df6c947010648f2aa8c01c2f5c07ed7" member_type: VOTER last_known_addr { host: "127.0.57.196" port: 41459 } } I0507 06:28:25.305411 1442 leader_election.cc:296] T 7d491c16d4ec46119cccd7f4c91f3df1 P 84c739a43bcf41b68638d0df1ae8e6ac [CANDIDATE]: Term 1 election: Requested vote from peers bc45a661e9494fc0bd570e9c5f333f05 (127.0.57.195:45153), 1df6c947010648f2aa8c01c2f5c07ed7 (127.0.57.196:41459) I0507 06:28:25.305811 1249 tablet_service.cc:1286] Received RequestConsensusVote() RPC: tablet_id: "7d491c16d4ec46119cccd7f4c91f3df1" candidate_uuid: "84c739a43bcf41b68638d0df1ae8e6ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc45a661e9494fc0bd570e9c5f333f05" I0507 06:28:25.305837 1371 tablet_service.cc:1286] Received RequestConsensusVote() RPC: tablet_id: "7d491c16d4ec46119cccd7f4c91f3df1" candidate_uuid: "84c739a43bcf41b68638d0df1ae8e6ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1df6c947010648f2aa8c01c2f5c07ed7" I0507 06:28:25.305958 1249 raft_consensus.cc:2886] T 7d491c16d4ec46119cccd7f4c91f3df1 P bc45a661e9494fc0bd570e9c5f333f05 [term 0 FOLLOWER]: Advancing to term 1 I0507 06:28:25.305989 1371 raft_consensus.cc:2886] T 7d491c16d4ec46119cccd7f4c91f3df1 P 1df6c947010648f2aa8c01c2f5c07ed7 [term 0 FOLLOWER]: Advancing to term 1 I0507 06:28:25.307601 1249 raft_consensus.cc:2346] T 7d491c16d4ec46119cccd7f4c91f3df1 P bc45a661e9494fc0bd570e9c5f333f05 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84c739a43bcf41b68638d0df1ae8e6ac in term 1. I0507 06:28:25.307612 1371 raft_consensus.cc:2346] T 7d491c16d4ec46119cccd7f4c91f3df1 P 1df6c947010648f2aa8c01c2f5c07ed7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84c739a43bcf41b68638d0df1ae8e6ac in term 1. I0507 06:28:25.308007 1058 leader_election.cc:310] T 7d491c16d4ec46119cccd7f4c91f3df1 P 84c739a43bcf41b68638d0df1ae8e6ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1df6c947010648f2aa8c01c2f5c07ed7, 84c739a43bcf41b68638d0df1ae8e6ac; no voters: I0507 06:28:25.308298 1442 raft_consensus.cc:2640] T 7d491c16d4ec46119cccd7f4c91f3df1 P 84c739a43bcf41b68638d0df1ae8e6ac [term 1 FOLLOWER]: Leader election won for term 1 I0507 06:28:25.308980 1442 raft_consensus.cc:667] T 7d491c16d4ec46119cccd7f4c91f3df1 P 84c739a43bcf41b68638d0df1ae8e6ac [term 1 LEADER]: Becoming Leader. State: Replica: 84c739a43bcf41b68638d0df1ae8e6ac, State: Running, Role: LEADER I0507 06:28:25.309406 1442 consensus_queue.cc:206] T 7d491c16d4ec46119cccd7f4c91f3df1 P 84c739a43bcf41b68638d0df1ae8e6ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc45a661e9494fc0bd570e9c5f333f05" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 45153 } } peers { permanent_uuid: "84c739a43bcf41b68638d0df1ae8e6ac" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 41441 } } peers { permanent_uuid: "1df6c947010648f2aa8c01c2f5c07ed7" member_type: VOTER last_known_addr { host: "127.0.57.196" port: 41459 } } I0507 06:28:25.313800 882 catalog_manager.cc:3988] T 7d491c16d4ec46119cccd7f4c91f3df1 P 84c739a43bcf41b68638d0df1ae8e6ac reported cstate change: term changed from 0 to 1, leader changed from to 84c739a43bcf41b68638d0df1ae8e6ac (127.0.57.194). New cstate: current_term: 1 leader_uuid: "84c739a43bcf41b68638d0df1ae8e6ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc45a661e9494fc0bd570e9c5f333f05" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 45153 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "84c739a43bcf41b68638d0df1ae8e6ac" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 41441 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1df6c947010648f2aa8c01c2f5c07ed7" member_type: VOTER last_known_addr { host: "127.0.57.196" port: 41459 } health_report { overall_health: UNKNOWN } } } I0507 06:28:25.334406 231 external_mini_cluster.cc:623] 4 TS(s) registered with all masters I0507 06:28:25.335893 231 ts_itest-base.cc:232] Waiting for 0 tablets on tserver 6109a76285154f0aaf369787c892238b to finish bootstrapping I0507 06:28:25.342748 231 ts_itest-base.cc:232] Waiting for 1 tablets on tserver 84c739a43bcf41b68638d0df1ae8e6ac to finish bootstrapping I0507 06:28:25.348578 231 ts_itest-base.cc:232] Waiting for 1 tablets on tserver bc45a661e9494fc0bd570e9c5f333f05 to finish bootstrapping I0507 06:28:25.354612 231 ts_itest-base.cc:232] Waiting for 1 tablets on tserver 1df6c947010648f2aa8c01c2f5c07ed7 to finish bootstrapping W0507 06:28:25.426236 1171 tablet.cc:1786] T 7d491c16d4ec46119cccd7f4c91f3df1 P 84c739a43bcf41b68638d0df1ae8e6ac: Can't schedule compaction. Clean time has not been advanced past its initial value. W0507 06:28:25.434306 1415 tablet.cc:1786] T 7d491c16d4ec46119cccd7f4c91f3df1 P 1df6c947010648f2aa8c01c2f5c07ed7: Can't schedule compaction. Clean time has not been advanced past its initial value. W0507 06:28:25.439513 1293 tablet.cc:1786] T 7d491c16d4ec46119cccd7f4c91f3df1 P bc45a661e9494fc0bd570e9c5f333f05: Can't schedule compaction. Clean time has not been advanced past its initial value. I0507 06:28:25.459209 1047 heartbeater.cc:467] Master 127.0.57.254:37367 was elected leader, sending a full tablet report... I0507 06:28:25.707957 1371 raft_consensus.cc:1181] T 7d491c16d4ec46119cccd7f4c91f3df1 P 1df6c947010648f2aa8c01c2f5c07ed7 [term 1 FOLLOWER]: Refusing update from remote peer 84c739a43bcf41b68638d0df1ae8e6ac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch) I0507 06:28:25.708863 1442 consensus_queue.cc:984] T 7d491c16d4ec46119cccd7f4c91f3df1 P 84c739a43bcf41b68638d0df1ae8e6ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "1df6c947010648f2aa8c01c2f5c07ed7" member_type: VOTER last_known_addr { host: "127.0.57.196" port: 41459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0507 06:28:25.721482 1249 raft_consensus.cc:1181] T 7d491c16d4ec46119cccd7f4c91f3df1 P bc45a661e9494fc0bd570e9c5f333f05 [term 1 FOLLOWER]: Refusing update from remote peer 84c739a43bcf41b68638d0df1ae8e6ac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch) I0507 06:28:25.722640 1452 consensus_queue.cc:984] T 7d491c16d4ec46119cccd7f4c91f3df1 P 84c739a43bcf41b68638d0df1ae8e6ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc45a661e9494fc0bd570e9c5f333f05" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 45153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0507 06:28:25.725729 1127 consensus_queue.cc:206] T 7d491c16d4ec46119cccd7f4c91f3df1 P 84c739a43bcf41b68638d0df1ae8e6ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bc45a661e9494fc0bd570e9c5f333f05" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 45153 } } peers { permanent_uuid: "84c739a43bcf41b68638d0df1ae8e6ac" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 41441 } } peers { permanent_uuid: "1df6c947010648f2aa8c01c2f5c07ed7" member_type: VOTER last_known_addr { host: "127.0.57.196" port: 41459 } } peers { permanent_uuid: "6109a76285154f0aaf369787c892238b" member_type: NON_VOTER last_known_addr { host: "127.0.57.193" port: 46143 } attrs { } } I0507 06:28:25.727612 1371 raft_consensus.cc:1181] T 7d491c16d4ec46119cccd7f4c91f3df1 P 1df6c947010648f2aa8c01c2f5c07ed7 [term 1 FOLLOWER]: Refusing update from remote peer 84c739a43bcf41b68638d0df1ae8e6ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0507 06:28:25.728097 1452 consensus_queue.cc:984] T 7d491c16d4ec46119cccd7f4c91f3df1 P 84c739a43bcf41b68638d0df1ae8e6ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "1df6c947010648f2aa8c01c2f5c07ed7" member_type: VOTER last_known_addr { host: "127.0.57.196" port: 41459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s I0507 06:28:25.730290 1445 raft_consensus.cc:2788] T 7d491c16d4ec46119cccd7f4c91f3df1 P 84c739a43bcf41b68638d0df1ae8e6ac [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6109a76285154f0aaf369787c892238b (127.0.57.193) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bc45a661e9494fc0bd570e9c5f333f05" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 45153 } } peers { permanent_uuid: "84c739a43bcf41b68638d0df1ae8e6ac" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 41441 } } peers { permanent_uuid: "1df6c947010648f2aa8c01c2f5c07ed7" member_type: VOTER last_known_addr { host: "127.0.57.196" port: 41459 } } peers { permanent_uuid: "6109a76285154f0aaf369787c892238b" member_type: NON_VOTER last_known_addr { host: "127.0.57.193" port: 46143 } attrs { } } } I0507 06:28:25.731007 1371 raft_consensus.cc:2788] T 7d491c16d4ec46119cccd7f4c91f3df1 P 1df6c947010648f2aa8c01c2f5c07ed7 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6109a76285154f0aaf369787c892238b (127.0.57.193) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bc45a661e9494fc0bd570e9c5f333f05" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 45153 } } peers { permanent_uuid: "84c739a43bcf41b68638d0df1ae8e6ac" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 41441 } } peers { permanent_uuid: "1df6c947010648f2aa8c01c2f5c07ed7" member_type: VOTER last_known_addr { host: "127.0.57.196" port: 41459 } } peers { permanent_uuid: "6109a76285154f0aaf369787c892238b" member_type: NON_VOTER last_known_addr { host: "127.0.57.193" port: 46143 } attrs { } } } W0507 06:28:25.734746 1059 consensus_peers.cc:457] T 7d491c16d4ec46119cccd7f4c91f3df1 P 84c739a43bcf41b68638d0df1ae8e6ac -> Peer 6109a76285154f0aaf369787c892238b (127.0.57.193:46143): Couldn't send request to peer 6109a76285154f0aaf369787c892238b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7d491c16d4ec46119cccd7f4c91f3df1. This is attempt 1: this message will repeat every 5th retry. I0507 06:28:25.735105 1248 raft_consensus.cc:1181] T 7d491c16d4ec46119cccd7f4c91f3df1 P bc45a661e9494fc0bd570e9c5f333f05 [term 1 FOLLOWER]: Refusing update from remote peer 84c739a43bcf41b68638d0df1ae8e6ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0507 06:28:25.735203 882 catalog_manager.cc:3988] T 7d491c16d4ec46119cccd7f4c91f3df1 P 84c739a43bcf41b68638d0df1ae8e6ac reported cstate change: config changed from index -1 to 2, NON_VOTER 6109a76285154f0aaf369787c892238b (127.0.57.193) added. New cstate: current_term: 1 leader_uuid: "84c739a43bcf41b68638d0df1ae8e6ac" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bc45a661e9494fc0bd570e9c5f333f05" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 45153 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "84c739a43bcf41b68638d0df1ae8e6ac" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 41441 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1df6c947010648f2aa8c01c2f5c07ed7" member_type: VOTER last_known_addr { host: "127.0.57.196" port: 41459 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6109a76285154f0aaf369787c892238b" member_type: NON_VOTER last_known_addr { host: "127.0.57.193" port: 46143 } attrs { } health_report { overall_health: UNKNOWN } } } I0507 06:28:25.735874 1442 consensus_queue.cc:984] T 7d491c16d4ec46119cccd7f4c91f3df1 P 84c739a43bcf41b68638d0df1ae8e6ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc45a661e9494fc0bd570e9c5f333f05" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 45153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s I0507 06:28:25.737320 1249 raft_consensus.cc:2788] T 7d491c16d4ec46119cccd7f4c91f3df1 P bc45a661e9494fc0bd570e9c5f333f05 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6109a76285154f0aaf369787c892238b (127.0.57.193) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bc45a661e9494fc0bd570e9c5f333f05" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 45153 } } peers { permanent_uuid: "84c739a43bcf41b68638d0df1ae8e6ac" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 41441 } } peers { permanent_uuid: "1df6c947010648f2aa8c01c2f5c07ed7" member_type: VOTER last_known_addr { host: "127.0.57.196" port: 41459 } } peers { permanent_uuid: "6109a76285154f0aaf369787c892238b" member_type: NON_VOTER last_known_addr { host: "127.0.57.193" port: 46143 } attrs { } } } I0507 06:28:26.113008 1455 ts_tablet_manager.cc:676] T 7d491c16d4ec46119cccd7f4c91f3df1 P 6109a76285154f0aaf369787c892238b: Initiating tablet copy from peer 84c739a43bcf41b68638d0df1ae8e6ac (127.0.57.194:41441) I0507 06:28:26.113343 1455 tablet_copy_client.cc:240] T 7d491c16d4ec46119cccd7f4c91f3df1 P 6109a76285154f0aaf369787c892238b: tablet copy: Beginning tablet copy session from remote peer at address 127.0.57.194:41441 I0507 06:28:26.118587 1147 tablet_copy_service.cc:135] P 84c739a43bcf41b68638d0df1ae8e6ac: Received BeginTabletCopySession request for tablet 7d491c16d4ec46119cccd7f4c91f3df1 from peer 6109a76285154f0aaf369787c892238b ({username='slave'} at 127.0.57.193:39317) I0507 06:28:26.118752 1147 tablet_copy_service.cc:156] P 84c739a43bcf41b68638d0df1ae8e6ac: Beginning new tablet copy session on tablet 7d491c16d4ec46119cccd7f4c91f3df1 from peer 6109a76285154f0aaf369787c892238b at {username='slave'} at 127.0.57.193:39317: session id = 6109a76285154f0aaf369787c892238b-7d491c16d4ec46119cccd7f4c91f3df1 I0507 06:28:26.121912 1147 tablet_copy_source_session.cc:202] T 7d491c16d4ec46119cccd7f4c91f3df1 P 84c739a43bcf41b68638d0df1ae8e6ac: Tablet Copy: opened 0 blocks and 1 log segments I0507 06:28:26.125499 1455 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet 7d491c16d4ec46119cccd7f4c91f3df1. 1 dirs total, 0 dirs full, 0 dirs failed I0507 06:28:26.134842 1455 tablet_copy_client.cc:575] T 7d491c16d4ec46119cccd7f4c91f3df1 P 6109a76285154f0aaf369787c892238b: tablet copy: Starting download of 0 data blocks... I0507 06:28:26.135033 1455 tablet_copy_client.cc:538] T 7d491c16d4ec46119cccd7f4c91f3df1 P 6109a76285154f0aaf369787c892238b: tablet copy: Starting download of 1 WAL segments... I0507 06:28:26.136986 1455 tablet_copy_client.cc:414] T 7d491c16d4ec46119cccd7f4c91f3df1 P 6109a76285154f0aaf369787c892238b: tablet copy: Tablet Copy complete. Replacing tablet superblock. I0507 06:28:26.139684 1455 tablet_bootstrap.cc:439] T 7d491c16d4ec46119cccd7f4c91f3df1 P 6109a76285154f0aaf369787c892238b: Bootstrap starting. I0507 06:28:26.144671 1455 log.cc:525] T 7d491c16d4ec46119cccd7f4c91f3df1 P 6109a76285154f0aaf369787c892238b: Log is configured to *not* fsync() on all Append() calls I0507 06:28:26.148042 1455 tablet_bootstrap.cc:439] T 7d491c16d4ec46119cccd7f4c91f3df1 P 6109a76285154f0aaf369787c892238b: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I0507 06:28:26.148373 1455 tablet_bootstrap.cc:439] T 7d491c16d4ec46119cccd7f4c91f3df1 P 6109a76285154f0aaf369787c892238b: Bootstrap complete. I0507 06:28:26.148674 1455 ts_tablet_manager.cc:1053] T 7d491c16d4ec46119cccd7f4c91f3df1 P 6109a76285154f0aaf369787c892238b: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.002s I0507 06:28:26.153960 1455 raft_consensus.cc:340] T 7d491c16d4ec46119cccd7f4c91f3df1 P 6109a76285154f0aaf369787c892238b [term 1 LEARNER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bc45a661e9494fc0bd570e9c5f333f05" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 45153 } } peers { permanent_uuid: "84c739a43bcf41b68638d0df1ae8e6ac" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 41441 } } peers { permanent_uuid: "1df6c947010648f2aa8c01c2f5c07ed7" member_type: VOTER last_known_addr { host: "127.0.57.196" port: 41459 } } peers { permanent_uuid: "6109a76285154f0aaf369787c892238b" member_type: NON_VOTER last_known_addr { host: "127.0.57.193" port: 46143 } attrs { } } I0507 06:28:26.154295 1455 raft_consensus.cc:705] T 7d491c16d4ec46119cccd7f4c91f3df1 P 6109a76285154f0aaf369787c892238b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6109a76285154f0aaf369787c892238b, State: Initialized, Role: LEARNER I0507 06:28:26.154598 1455 consensus_queue.cc:229] T 7d491c16d4ec46119cccd7f4c91f3df1 P 6109a76285154f0aaf369787c892238b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bc45a661e9494fc0bd570e9c5f333f05" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 45153 } } peers { permanent_uuid: "84c739a43bcf41b68638d0df1ae8e6ac" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 41441 } } peers { permanent_uuid: "1df6c947010648f2aa8c01c2f5c07ed7" member_type: VOTER last_known_addr { host: "127.0.57.196" port: 41459 } } peers { permanent_uuid: "6109a76285154f0aaf369787c892238b" member_type: NON_VOTER last_known_addr { host: "127.0.57.193" port: 46143 } attrs { } } I0507 06:28:26.156030 1455 ts_tablet_manager.cc:1081] T 7d491c16d4ec46119cccd7f4c91f3df1 P 6109a76285154f0aaf369787c892238b: Time spent starting tablet: real 0.007s user 0.005s sys 0.001s I0507 06:28:26.156880 1147 tablet_copy_service.cc:325] P 84c739a43bcf41b68638d0df1ae8e6ac: Request end of tablet copy session 6109a76285154f0aaf369787c892238b-7d491c16d4ec46119cccd7f4c91f3df1 received from {username='slave'} at 127.0.57.193:39317 I0507 06:28:26.157002 1147 tablet_copy_service.cc:413] P 84c739a43bcf41b68638d0df1ae8e6ac: ending tablet copy session 6109a76285154f0aaf369787c892238b-7d491c16d4ec46119cccd7f4c91f3df1 on tablet 7d491c16d4ec46119cccd7f4c91f3df1 with peer 6109a76285154f0aaf369787c892238b I0507 06:28:26.190958 231 external_mini_cluster.cc:1172] Attempting to check leaks for /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver pid 918 I0507 06:28:26.191769 1044 generic_service.cc:171] Checking for leaks (request via RPC) I0507 06:28:26.247726 231 external_mini_cluster.cc:1131] Killing /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver with pid 918 I0507 06:28:26.257891 231 external_mini_cluster.cc:1172] Attempting to check leaks for /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver pid 1051 I0507 06:28:26.258769 1167 generic_service.cc:171] Checking for leaks (request via RPC) I0507 06:28:26.308570 231 external_mini_cluster.cc:1131] Killing /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver with pid 1051 I0507 06:28:26.318517 231 external_mini_cluster.cc:1172] Attempting to check leaks for /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver pid 1173 I0507 06:28:26.319466 1289 generic_service.cc:171] Checking for leaks (request via RPC) I0507 06:28:26.366073 231 external_mini_cluster.cc:1131] Killing /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver with pid 1173 I0507 06:28:26.376313 231 external_mini_cluster.cc:1172] Attempting to check leaks for /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver pid 1295 I0507 06:28:26.377672 1411 generic_service.cc:171] Checking for leaks (request via RPC) I0507 06:28:26.424136 231 external_mini_cluster.cc:1131] Killing /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver with pid 1295 I0507 06:28:26.434381 231 external_mini_cluster.cc:1172] Attempting to check leaks for /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-master pid 858 I0507 06:28:26.435380 914 generic_service.cc:171] Checking for leaks (request via RPC) I0507 06:28:26.508661 231 external_mini_cluster.cc:1131] Killing /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-master with pid 858 [ OK ] RaftConsensusNonVoterITest.ReplicaMatchPolicy (2883 ms) [ RUN ] RaftConsensusNonVoterITest.AddNonVoterReplica I0507 06:28:26.532300 231 test_util.cc:212] Using random seed: 1393890741 I0507 06:28:26.533850 231 ts_itest-base.cc:112] Starting cluster with: I0507 06:28:26.533882 231 ts_itest-base.cc:113] -------------- I0507 06:28:26.533907 231 ts_itest-base.cc:114] 3 tablet servers I0507 06:28:26.533936 231 ts_itest-base.cc:115] 2 replicas per TS I0507 06:28:26.533962 231 ts_itest-base.cc:116] -------------- I0507 06:28:26.535109 231 external_mini_cluster.cc:933] Running /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-master /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-master --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.AddNonVoterReplica.1557210500371863-231/raft_consensus-itest-cluster/master-0/wal --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.AddNonVoterReplica.1557210500371863-231/raft_consensus-itest-cluster/master-0/data --block_manager=log --webserver_interface=localhost --ipki_ca_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.0.57.254:38587 --webserver_interface=127.0.57.254 --webserver_port=0 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.AddNonVoterReplica.1557210500371863-231/raft_consensus-itest-cluster/master-0/logs --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.AddNonVoterReplica.1557210500371863-231/raft_consensus-itest-cluster/master-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --allow_unsafe_replication_factor=true --catalog_manager_evict_excess_replicas=false --rpc_reuseport=true with env {} WARNING: Logging before InitGoogleLogging() is written to STDERR W0507 06:28:26.724647 1476 flags.cc:404] Enabled unsafe flag: --allow_unsafe_replication_factor=true W0507 06:28:26.724877 1476 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0507 06:28:26.724946 1476 flags.cc:404] Enabled unsafe flag: --never_fsync=true W0507 06:28:26.727396 1476 flags.cc:404] Enabled experimental flag: --ipki_ca_key_size=1024 W0507 06:28:26.727444 1476 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W0507 06:28:26.727463 1476 flags.cc:404] Enabled experimental flag: --tsk_num_rsa_bits=512 W0507 06:28:26.727502 1476 flags.cc:404] Enabled experimental flag: --rpc_reuseport=true I0507 06:28:26.731957 1476 master_main.cc:73] Master server non-default flags: --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.AddNonVoterReplica.1557210500371863-231/raft_consensus-itest-cluster/master-0/data --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.AddNonVoterReplica.1557210500371863-231/raft_consensus-itest-cluster/master-0/wal --allow_unsafe_replication_factor=true --catalog_manager_evict_excess_replicas=false --ipki_ca_key_size=1024 --ipki_server_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.0.57.254:38587 --rpc_reuseport=true --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.AddNonVoterReplica.1557210500371863-231/raft_consensus-itest-cluster/master-0/data/info.pb --webserver_interface=127.0.57.254 --webserver_port=0 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.AddNonVoterReplica.1557210500371863-231/raft_consensus-itest-cluster/master-0/logs --logbuflevel=-1 --logtostderr=true Master server version: kudu 1.10.0-cdh6.x-SNAPSHOT revision 7d0ee26697926e87af9ef395008799577f6ab43f build type FASTDEBUG built by jenkins at 06 May 2019 22:44:21 PST on quasar-knmpfx-1.vpc.cloudera.com build id 322017 ASAN enabled I0507 06:28:26.738549 1476 system_ntp.cc:202] NTP initialized. Skew: 500ppm Current error: 27516us I0507 06:28:26.739807 1476 fs_manager.cc:263] Metadata directory not provided I0507 06:28:26.739866 1476 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0507 06:28:26.740167 1476 server_base.cc:433] Could not load existing FS layout: Not found: could not find a healthy instance file I0507 06:28:26.740227 1476 server_base.cc:434] Attempting to create new FS layout instead I0507 06:28:26.742672 1476 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.AddNonVoterReplica.1557210500371863-231/raft_consensus-itest-cluster/master-0/data/instance: uuid: "4365e3a555824c77be1797521bce6de6" format_stamp: "Formatted at 2019-05-07 06:28:26 on dist-test-slave-dist-test-slave-rqp6" I0507 06:28:26.743168 1476 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.AddNonVoterReplica.1557210500371863-231/raft_consensus-itest-cluster/master-0/wal/instance: uuid: "4365e3a555824c77be1797521bce6de6" format_stamp: "Formatted at 2019-05-07 06:28:26 on dist-test-slave-dist-test-slave-rqp6" I0507 06:28:26.749758 1476 fs_manager.cc:503] Time spent creating directory manager: real 0.006s user 0.002s sys 0.005s I0507 06:28:26.750967 1476 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0507 06:28:26.751531 1476 file_cache.cc:466] Constructed file cache lbm with capacity 419430 I0507 06:28:26.754593 1476 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s I0507 06:28:26.754663 1476 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.AddNonVoterReplica.1557210500371863-231/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.AddNonVoterReplica.1557210500371863-231/raft_consensus-itest-cluster/master-0/wal uuid: "4365e3a555824c77be1797521bce6de6" format_stamp: "Formatted at 2019-05-07 06:28:26 on dist-test-slave-dist-test-slave-rqp6" I0507 06:28:26.754819 1476 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.AddNonVoterReplica.1557210500371863-231/raft_consensus-itest-cluster/master-0/wal metadata directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.AddNonVoterReplica.1557210500371863-231/raft_consensus-itest-cluster/master-0/wal 1 data directories: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.AddNonVoterReplica.1557210500371863-231/raft_consensus-itest-cluster/master-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0507 06:28:26.786022 1476 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0507 06:28:26.798507 1476 rpc_server.cc:205] RPC server started. Bound to: 127.0.57.254:38587 I0507 06:28:26.799553 1476 webserver.cc:308] Webserver started at http://127.0.57.254:41569/ using document root and password file I0507 06:28:26.800256 1476 server_base.cc:608] Dumped server information to /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.AddNonVoterReplica.1557210500371863-231/raft_consensus-itest-cluster/master-0/data/info.pb I0507 06:28:26.803679 1535 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0507 06:28:26.807620 231 external_mini_cluster.cc:995] Started /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-master as pid 1476 I0507 06:28:26.808308 231 external_mini_cluster.cc:933] Running /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.AddNonVoterReplica.1557210500371863-231/raft_consensus-itest-cluster/ts-0/wal --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.AddNonVoterReplica.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data --block_manager=log --rpc_bind_addresses=127.0.57.193:0 --local_ip_for_outbound_sockets=127.0.57.193 --webserver_interface=127.0.57.193 --webserver_port=0 --tserver_master_addrs=127.0.57.254:38587 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.AddNonVoterReplica.1557210500371863-231/raft_consensus-itest-cluster/ts-0/logs --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.AddNonVoterReplica.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --tablet_copy_download_file_inject_latency_ms=1000 with env {} I0507 06:28:26.815140 1535 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 4365e3a555824c77be1797521bce6de6: Bootstrap starting. I0507 06:28:26.818403 1535 tablet_bootstrap.cc:590] T 00000000000000000000000000000000 P 4365e3a555824c77be1797521bce6de6: No blocks or log segments found. Creating new log. I0507 06:28:26.819507 1535 log.cc:525] T 00000000000000000000000000000000 P 4365e3a555824c77be1797521bce6de6: Log is configured to *not* fsync() on all Append() calls I0507 06:28:26.821393 1535 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 4365e3a555824c77be1797521bce6de6: No bootstrap required, opened a new log I0507 06:28:26.830247 1535 raft_consensus.cc:340] T 00000000000000000000000000000000 P 4365e3a555824c77be1797521bce6de6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4365e3a555824c77be1797521bce6de6" member_type: VOTER } I0507 06:28:26.830514 1535 raft_consensus.cc:366] T 00000000000000000000000000000000 P 4365e3a555824c77be1797521bce6de6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0507 06:28:26.830579 1535 raft_consensus.cc:705] T 00000000000000000000000000000000 P 4365e3a555824c77be1797521bce6de6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4365e3a555824c77be1797521bce6de6, State: Initialized, Role: FOLLOWER I0507 06:28:26.831190 1535 consensus_queue.cc:229] T 00000000000000000000000000000000 P 4365e3a555824c77be1797521bce6de6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4365e3a555824c77be1797521bce6de6" member_type: VOTER } I0507 06:28:26.831400 1535 raft_consensus.cc:380] T 00000000000000000000000000000000 P 4365e3a555824c77be1797521bce6de6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0507 06:28:26.831490 1535 raft_consensus.cc:472] T 00000000000000000000000000000000 P 4365e3a555824c77be1797521bce6de6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0507 06:28:26.831604 1535 raft_consensus.cc:2886] T 00000000000000000000000000000000 P 4365e3a555824c77be1797521bce6de6 [term 0 FOLLOWER]: Advancing to term 1 I0507 06:28:26.833806 1535 raft_consensus.cc:494] T 00000000000000000000000000000000 P 4365e3a555824c77be1797521bce6de6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4365e3a555824c77be1797521bce6de6" member_type: VOTER } I0507 06:28:26.834313 1535 leader_election.cc:310] T 00000000000000000000000000000000 P 4365e3a555824c77be1797521bce6de6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4365e3a555824c77be1797521bce6de6; no voters: I0507 06:28:26.835160 1538 raft_consensus.cc:2640] T 00000000000000000000000000000000 P 4365e3a555824c77be1797521bce6de6 [term 1 FOLLOWER]: Leader election won for term 1 I0507 06:28:26.835537 1535 leader_election.cc:296] T 00000000000000000000000000000000 P 4365e3a555824c77be1797521bce6de6 [CANDIDATE]: Term 1 election: Requested vote from peers I0507 06:28:26.836179 1538 raft_consensus.cc:667] T 00000000000000000000000000000000 P 4365e3a555824c77be1797521bce6de6 [term 1 LEADER]: Becoming Leader. State: Replica: 4365e3a555824c77be1797521bce6de6, State: Running, Role: LEADER I0507 06:28:26.836720 1538 consensus_queue.cc:206] T 00000000000000000000000000000000 P 4365e3a555824c77be1797521bce6de6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4365e3a555824c77be1797521bce6de6" member_type: VOTER } I0507 06:28:26.840592 1535 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4365e3a555824c77be1797521bce6de6 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 1 leader_uuid: "4365e3a555824c77be1797521bce6de6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4365e3a555824c77be1797521bce6de6" member_type: VOTER } } I0507 06:28:26.841022 1535 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4365e3a555824c77be1797521bce6de6 [sys.catalog]: This master's current role is: LEADER I0507 06:28:26.840750 1539 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4365e3a555824c77be1797521bce6de6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4365e3a555824c77be1797521bce6de6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4365e3a555824c77be1797521bce6de6" member_type: VOTER } } I0507 06:28:26.841722 1539 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4365e3a555824c77be1797521bce6de6 [sys.catalog]: This master's current role is: LEADER I0507 06:28:26.841262 1540 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4365e3a555824c77be1797521bce6de6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4365e3a555824c77be1797521bce6de6. Latest consensus state: current_term: 1 leader_uuid: "4365e3a555824c77be1797521bce6de6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4365e3a555824c77be1797521bce6de6" member_type: VOTER } } I0507 06:28:26.841814 1540 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4365e3a555824c77be1797521bce6de6 [sys.catalog]: This master's current role is: LEADER I0507 06:28:26.843503 1535 sys_catalog.cc:424] T 00000000000000000000000000000000 P 4365e3a555824c77be1797521bce6de6 [sys.catalog]: configured and running, proceeding with master startup. I0507 06:28:26.843729 1544 catalog_manager.cc:1055] Loading table and tablet metadata into memory... I0507 06:28:26.848337 1544 catalog_manager.cc:1066] Initializing Kudu internal certificate authority... I0507 06:28:26.894307 1544 catalog_manager.cc:950] Generated new certificate authority record I0507 06:28:26.896216 1544 catalog_manager.cc:1075] Loading token signing keys... I0507 06:28:26.913944 1544 catalog_manager.cc:4292] T 00000000000000000000000000000000 P 4365e3a555824c77be1797521bce6de6: Generated new TSK 0 WARNING: Logging before InitGoogleLogging() is written to STDERR W0507 06:28:26.969060 1536 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0507 06:28:26.969308 1536 flags.cc:404] Enabled unsafe flag: --never_fsync=true W0507 06:28:26.971417 1536 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W0507 06:28:26.971535 1536 flags.cc:404] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.57.193 I0507 06:28:26.988739 1536 tablet_server_main.cc:78] Tablet server non-default flags: --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.AddNonVoterReplica.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.AddNonVoterReplica.1557210500371863-231/raft_consensus-itest-cluster/ts-0/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.0.57.193:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.AddNonVoterReplica.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data/info.pb --webserver_interface=127.0.57.193 --webserver_port=0 --tablet_copy_download_file_inject_latency_ms=1000 --tserver_master_addrs=127.0.57.254:38587 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.0.57.193 --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.AddNonVoterReplica.1557210500371863-231/raft_consensus-itest-cluster/ts-0/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.10.0-cdh6.x-SNAPSHOT revision 7d0ee26697926e87af9ef395008799577f6ab43f build type FASTDEBUG built by jenkins at 06 May 2019 22:44:21 PST on quasar-knmpfx-1.vpc.cloudera.com build id 322017 ASAN enabled I0507 06:28:26.996724 1536 system_ntp.cc:202] NTP initialized. Skew: 500ppm Current error: 27516us I0507 06:28:26.997372 1536 fs_manager.cc:263] Metadata directory not provided I0507 06:28:26.997452 1536 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0507 06:28:26.997716 1536 server_base.cc:433] Could not load existing FS layout: Not found: could not find a healthy instance file I0507 06:28:26.997792 1536 server_base.cc:434] Attempting to create new FS layout instead I0507 06:28:27.000161 1536 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.AddNonVoterReplica.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data/instance: uuid: "180459af951d4aabbb5312ba1ae741bb" format_stamp: "Formatted at 2019-05-07 06:28:26 on dist-test-slave-dist-test-slave-rqp6" I0507 06:28:27.000782 1536 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.AddNonVoterReplica.1557210500371863-231/raft_consensus-itest-cluster/ts-0/wal/instance: uuid: "180459af951d4aabbb5312ba1ae741bb" format_stamp: "Formatted at 2019-05-07 06:28:26 on dist-test-slave-dist-test-slave-rqp6" I0507 06:28:27.009301 1536 fs_manager.cc:503] Time spent creating directory manager: real 0.008s user 0.006s sys 0.001s I0507 06:28:27.010341 1536 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0507 06:28:27.010933 1536 file_cache.cc:466] Constructed file cache lbm with capacity 419430 I0507 06:28:27.013835 1536 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s I0507 06:28:27.013901 1536 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.AddNonVoterReplica.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.AddNonVoterReplica.1557210500371863-231/raft_consensus-itest-cluster/ts-0/wal uuid: "180459af951d4aabbb5312ba1ae741bb" format_stamp: "Formatted at 2019-05-07 06:28:26 on dist-test-slave-dist-test-slave-rqp6" I0507 06:28:27.014056 1536 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.AddNonVoterReplica.1557210500371863-231/raft_consensus-itest-cluster/ts-0/wal metadata directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.AddNonVoterReplica.1557210500371863-231/raft_consensus-itest-cluster/ts-0/wal 1 data directories: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.AddNonVoterReplica.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0507 06:28:27.061561 1536 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0507 06:28:27.073684 1536 ts_tablet_manager.cc:356] Loaded tablet metadata (0 total tablets, 0 live tablets) I0507 06:28:27.073750 1536 ts_tablet_manager.cc:376] Registered 0 tablets I0507 06:28:27.220021 1536 rpc_server.cc:205] RPC server started. Bound to: 127.0.57.193:37035 I0507 06:28:27.221212 1536 webserver.cc:308] Webserver started at http://127.0.57.193:33703/ using document root and password file I0507 06:28:27.221916 1536 server_base.cc:608] Dumped server information to /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.AddNonVoterReplica.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data/info.pb I0507 06:28:27.224910 231 external_mini_cluster.cc:995] Started /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver as pid 1536 I0507 06:28:27.225512 231 external_mini_cluster.cc:933] Running /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.AddNonVoterReplica.1557210500371863-231/raft_consensus-itest-cluster/ts-1/wal --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.AddNonVoterReplica.1557210500371863-231/raft_consensus-itest-cluster/ts-1/data --block_manager=log --rpc_bind_addresses=127.0.57.194:0 --local_ip_for_outbound_sockets=127.0.57.194 --webserver_interface=127.0.57.194 --webserver_port=0 --tserver_master_addrs=127.0.57.254:38587 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.AddNonVoterReplica.1557210500371863-231/raft_consensus-itest-cluster/ts-1/logs --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.AddNonVoterReplica.1557210500371863-231/raft_consensus-itest-cluster/ts-1/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --tablet_copy_download_file_inject_latency_ms=1000 with env {} I0507 06:28:27.241726 1666 heartbeater.cc:345] Connected to a master server at 127.0.57.254:38587 I0507 06:28:27.241925 1666 heartbeater.cc:426] Registering TS with master... I0507 06:28:27.242439 1666 heartbeater.cc:475] Master 127.0.57.254:38587 requested a full tablet report, sending... I0507 06:28:27.243943 1502 ts_manager.cc:159] Registered new tserver with Master: 180459af951d4aabbb5312ba1ae741bb (127.0.57.193:37035) I0507 06:28:27.245493 1502 master_service.cc:273] Signed X509 certificate for tserver {username='slave'} at 127.0.57.193:38159 WARNING: Logging before InitGoogleLogging() is written to STDERR W0507 06:28:27.375949 1670 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0507 06:28:27.376200 1670 flags.cc:404] Enabled unsafe flag: --never_fsync=true W0507 06:28:27.378388 1670 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W0507 06:28:27.378518 1670 flags.cc:404] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.57.194 I0507 06:28:27.382364 1670 tablet_server_main.cc:78] Tablet server non-default flags: --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.AddNonVoterReplica.1557210500371863-231/raft_consensus-itest-cluster/ts-1/data --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.AddNonVoterReplica.1557210500371863-231/raft_consensus-itest-cluster/ts-1/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.0.57.194:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.AddNonVoterReplica.1557210500371863-231/raft_consensus-itest-cluster/ts-1/data/info.pb --webserver_interface=127.0.57.194 --webserver_port=0 --tablet_copy_download_file_inject_latency_ms=1000 --tserver_master_addrs=127.0.57.254:38587 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.0.57.194 --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.AddNonVoterReplica.1557210500371863-231/raft_consensus-itest-cluster/ts-1/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.10.0-cdh6.x-SNAPSHOT revision 7d0ee26697926e87af9ef395008799577f6ab43f build type FASTDEBUG built by jenkins at 06 May 2019 22:44:21 PST on quasar-knmpfx-1.vpc.cloudera.com build id 322017 ASAN enabled I0507 06:28:27.389642 1670 system_ntp.cc:202] NTP initialized. Skew: 500ppm Current error: 28016us I0507 06:28:27.390265 1670 fs_manager.cc:263] Metadata directory not provided I0507 06:28:27.390342 1670 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0507 06:28:27.390612 1670 server_base.cc:433] Could not load existing FS layout: Not found: could not find a healthy instance file I0507 06:28:27.390684 1670 server_base.cc:434] Attempting to create new FS layout instead I0507 06:28:27.392953 1670 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.AddNonVoterReplica.1557210500371863-231/raft_consensus-itest-cluster/ts-1/data/instance: uuid: "af7bb239a5d6454daa140682a17b532d" format_stamp: "Formatted at 2019-05-07 06:28:27 on dist-test-slave-dist-test-slave-rqp6" I0507 06:28:27.393501 1670 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.AddNonVoterReplica.1557210500371863-231/raft_consensus-itest-cluster/ts-1/wal/instance: uuid: "af7bb239a5d6454daa140682a17b532d" format_stamp: "Formatted at 2019-05-07 06:28:27 on dist-test-slave-dist-test-slave-rqp6" I0507 06:28:27.399518 1670 fs_manager.cc:503] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s I0507 06:28:27.400475 1670 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0507 06:28:27.401028 1670 file_cache.cc:466] Constructed file cache lbm with capacity 419430 I0507 06:28:27.403767 1670 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s I0507 06:28:27.403828 1670 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.AddNonVoterReplica.1557210500371863-231/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.AddNonVoterReplica.1557210500371863-231/raft_consensus-itest-cluster/ts-1/wal uuid: "af7bb239a5d6454daa140682a17b532d" format_stamp: "Formatted at 2019-05-07 06:28:27 on dist-test-slave-dist-test-slave-rqp6" I0507 06:28:27.403975 1670 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.AddNonVoterReplica.1557210500371863-231/raft_consensus-itest-cluster/ts-1/wal metadata directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.AddNonVoterReplica.1557210500371863-231/raft_consensus-itest-cluster/ts-1/wal 1 data directories: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.AddNonVoterReplica.1557210500371863-231/raft_consensus-itest-cluster/ts-1/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0507 06:28:27.465314 1670 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0507 06:28:27.466971 1670 ts_tablet_manager.cc:356] Loaded tablet metadata (0 total tablets, 0 live tablets) I0507 06:28:27.467031 1670 ts_tablet_manager.cc:376] Registered 0 tablets I0507 06:28:27.488898 1670 rpc_server.cc:205] RPC server started. Bound to: 127.0.57.194:43367 I0507 06:28:27.489992 1670 webserver.cc:308] Webserver started at http://127.0.57.194:42655/ using document root and password file I0507 06:28:27.490682 1670 server_base.cc:608] Dumped server information to /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.AddNonVoterReplica.1557210500371863-231/raft_consensus-itest-cluster/ts-1/data/info.pb I0507 06:28:27.498245 231 external_mini_cluster.cc:995] Started /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver as pid 1670 I0507 06:28:27.498982 231 external_mini_cluster.cc:933] Running /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.AddNonVoterReplica.1557210500371863-231/raft_consensus-itest-cluster/ts-2/wal --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.AddNonVoterReplica.1557210500371863-231/raft_consensus-itest-cluster/ts-2/data --block_manager=log --rpc_bind_addresses=127.0.57.195:0 --local_ip_for_outbound_sockets=127.0.57.195 --webserver_interface=127.0.57.195 --webserver_port=0 --tserver_master_addrs=127.0.57.254:38587 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.AddNonVoterReplica.1557210500371863-231/raft_consensus-itest-cluster/ts-2/logs --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.AddNonVoterReplica.1557210500371863-231/raft_consensus-itest-cluster/ts-2/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --tablet_copy_download_file_inject_latency_ms=1000 with env {} I0507 06:28:27.503171 1789 heartbeater.cc:345] Connected to a master server at 127.0.57.254:38587 I0507 06:28:27.503379 1789 heartbeater.cc:426] Registering TS with master... I0507 06:28:27.503964 1789 heartbeater.cc:475] Master 127.0.57.254:38587 requested a full tablet report, sending... I0507 06:28:27.505228 1502 ts_manager.cc:159] Registered new tserver with Master: af7bb239a5d6454daa140682a17b532d (127.0.57.194:43367) I0507 06:28:27.506429 1502 master_service.cc:273] Signed X509 certificate for tserver {username='slave'} at 127.0.57.194:36239 WARNING: Logging before InitGoogleLogging() is written to STDERR W0507 06:28:27.650122 1792 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0507 06:28:27.650334 1792 flags.cc:404] Enabled unsafe flag: --never_fsync=true W0507 06:28:27.652354 1792 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W0507 06:28:27.652446 1792 flags.cc:404] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.57.195 I0507 06:28:27.657423 1792 tablet_server_main.cc:78] Tablet server non-default flags: --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.AddNonVoterReplica.1557210500371863-231/raft_consensus-itest-cluster/ts-2/data --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.AddNonVoterReplica.1557210500371863-231/raft_consensus-itest-cluster/ts-2/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.0.57.195:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.AddNonVoterReplica.1557210500371863-231/raft_consensus-itest-cluster/ts-2/data/info.pb --webserver_interface=127.0.57.195 --webserver_port=0 --tablet_copy_download_file_inject_latency_ms=1000 --tserver_master_addrs=127.0.57.254:38587 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.0.57.195 --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.AddNonVoterReplica.1557210500371863-231/raft_consensus-itest-cluster/ts-2/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.10.0-cdh6.x-SNAPSHOT revision 7d0ee26697926e87af9ef395008799577f6ab43f build type FASTDEBUG built by jenkins at 06 May 2019 22:44:21 PST on quasar-knmpfx-1.vpc.cloudera.com build id 322017 ASAN enabled I0507 06:28:27.664821 1792 system_ntp.cc:202] NTP initialized. Skew: 500ppm Current error: 28016us I0507 06:28:27.665443 1792 fs_manager.cc:263] Metadata directory not provided I0507 06:28:27.665508 1792 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0507 06:28:27.665755 1792 server_base.cc:433] Could not load existing FS layout: Not found: could not find a healthy instance file I0507 06:28:27.665819 1792 server_base.cc:434] Attempting to create new FS layout instead I0507 06:28:27.668081 1792 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.AddNonVoterReplica.1557210500371863-231/raft_consensus-itest-cluster/ts-2/data/instance: uuid: "f9f586dd86b04fe984968b9de7f52209" format_stamp: "Formatted at 2019-05-07 06:28:27 on dist-test-slave-dist-test-slave-rqp6" I0507 06:28:27.668643 1792 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.AddNonVoterReplica.1557210500371863-231/raft_consensus-itest-cluster/ts-2/wal/instance: uuid: "f9f586dd86b04fe984968b9de7f52209" format_stamp: "Formatted at 2019-05-07 06:28:27 on dist-test-slave-dist-test-slave-rqp6" I0507 06:28:27.674726 1792 fs_manager.cc:503] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s I0507 06:28:27.675717 1792 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0507 06:28:27.676281 1792 file_cache.cc:466] Constructed file cache lbm with capacity 419430 I0507 06:28:27.679044 1792 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s I0507 06:28:27.679111 1792 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.AddNonVoterReplica.1557210500371863-231/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.AddNonVoterReplica.1557210500371863-231/raft_consensus-itest-cluster/ts-2/wal uuid: "f9f586dd86b04fe984968b9de7f52209" format_stamp: "Formatted at 2019-05-07 06:28:27 on dist-test-slave-dist-test-slave-rqp6" I0507 06:28:27.679250 1792 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.AddNonVoterReplica.1557210500371863-231/raft_consensus-itest-cluster/ts-2/wal metadata directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.AddNonVoterReplica.1557210500371863-231/raft_consensus-itest-cluster/ts-2/wal 1 data directories: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.AddNonVoterReplica.1557210500371863-231/raft_consensus-itest-cluster/ts-2/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0507 06:28:27.695785 1792 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0507 06:28:27.697388 1792 ts_tablet_manager.cc:356] Loaded tablet metadata (0 total tablets, 0 live tablets) I0507 06:28:27.697443 1792 ts_tablet_manager.cc:376] Registered 0 tablets I0507 06:28:27.718924 1792 rpc_server.cc:205] RPC server started. Bound to: 127.0.57.195:36235 I0507 06:28:27.719975 1792 webserver.cc:308] Webserver started at http://127.0.57.195:37103/ using document root and password file I0507 06:28:27.720644 1792 server_base.cc:608] Dumped server information to /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.AddNonVoterReplica.1557210500371863-231/raft_consensus-itest-cluster/ts-2/data/info.pb I0507 06:28:27.721735 231 external_mini_cluster.cc:995] Started /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver as pid 1792 I0507 06:28:27.738684 1911 heartbeater.cc:345] Connected to a master server at 127.0.57.254:38587 I0507 06:28:27.738904 1911 heartbeater.cc:426] Registering TS with master... I0507 06:28:27.739428 1911 heartbeater.cc:475] Master 127.0.57.254:38587 requested a full tablet report, sending... I0507 06:28:27.740684 1502 ts_manager.cc:159] Registered new tserver with Master: f9f586dd86b04fe984968b9de7f52209 (127.0.57.195:36235) I0507 06:28:27.741019 231 external_mini_cluster.cc:623] 3 TS(s) registered with all masters I0507 06:28:27.741858 1502 master_service.cc:273] Signed X509 certificate for tserver {username='slave'} at 127.0.57.195:49463 I0507 06:28:27.751257 1502 catalog_manager.cc:1415] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41512: name: "TestTable" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 2 split_rows_range_bounds { } partition_schema { range_schema { columns { name: "key" } } } I0507 06:28:27.782274 1726 tablet_service.cc:1016] Processing CreateTablet for tablet 9d26ac1dc0994a898fbb8aef9162ee61 (table=TestTable [id=8751e1e9d0db481bbcecc22d87ad6975]), partition=RANGE (key) PARTITION UNBOUNDED I0507 06:28:27.783072 1726 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet 9d26ac1dc0994a898fbb8aef9162ee61. 1 dirs total, 0 dirs full, 0 dirs failed I0507 06:28:27.780174 1848 tablet_service.cc:1016] Processing CreateTablet for tablet 9d26ac1dc0994a898fbb8aef9162ee61 (table=TestTable [id=8751e1e9d0db481bbcecc22d87ad6975]), partition=RANGE (key) PARTITION UNBOUNDED I0507 06:28:27.785126 1848 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet 9d26ac1dc0994a898fbb8aef9162ee61. 1 dirs total, 0 dirs full, 0 dirs failed I0507 06:28:27.802799 1928 tablet_bootstrap.cc:439] T 9d26ac1dc0994a898fbb8aef9162ee61 P af7bb239a5d6454daa140682a17b532d: Bootstrap starting. I0507 06:28:27.804998 1929 tablet_bootstrap.cc:439] T 9d26ac1dc0994a898fbb8aef9162ee61 P f9f586dd86b04fe984968b9de7f52209: Bootstrap starting. I0507 06:28:27.806998 1929 tablet_bootstrap.cc:590] T 9d26ac1dc0994a898fbb8aef9162ee61 P f9f586dd86b04fe984968b9de7f52209: No blocks or log segments found. Creating new log. I0507 06:28:27.807837 1929 log.cc:525] T 9d26ac1dc0994a898fbb8aef9162ee61 P f9f586dd86b04fe984968b9de7f52209: Log is configured to *not* fsync() on all Append() calls I0507 06:28:27.809310 1928 tablet_bootstrap.cc:590] T 9d26ac1dc0994a898fbb8aef9162ee61 P af7bb239a5d6454daa140682a17b532d: No blocks or log segments found. Creating new log. I0507 06:28:27.810139 1928 log.cc:525] T 9d26ac1dc0994a898fbb8aef9162ee61 P af7bb239a5d6454daa140682a17b532d: Log is configured to *not* fsync() on all Append() calls I0507 06:28:27.811600 1928 tablet_bootstrap.cc:439] T 9d26ac1dc0994a898fbb8aef9162ee61 P af7bb239a5d6454daa140682a17b532d: No bootstrap required, opened a new log I0507 06:28:27.811605 1929 tablet_bootstrap.cc:439] T 9d26ac1dc0994a898fbb8aef9162ee61 P f9f586dd86b04fe984968b9de7f52209: No bootstrap required, opened a new log I0507 06:28:27.811823 1929 ts_tablet_manager.cc:1053] T 9d26ac1dc0994a898fbb8aef9162ee61 P f9f586dd86b04fe984968b9de7f52209: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s I0507 06:28:27.811825 1928 ts_tablet_manager.cc:1053] T 9d26ac1dc0994a898fbb8aef9162ee61 P af7bb239a5d6454daa140682a17b532d: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.000s I0507 06:28:27.817173 1928 raft_consensus.cc:340] T 9d26ac1dc0994a898fbb8aef9162ee61 P af7bb239a5d6454daa140682a17b532d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9f586dd86b04fe984968b9de7f52209" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 36235 } } peers { permanent_uuid: "af7bb239a5d6454daa140682a17b532d" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 43367 } } I0507 06:28:27.817147 1929 raft_consensus.cc:340] T 9d26ac1dc0994a898fbb8aef9162ee61 P f9f586dd86b04fe984968b9de7f52209 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9f586dd86b04fe984968b9de7f52209" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 36235 } } peers { permanent_uuid: "af7bb239a5d6454daa140682a17b532d" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 43367 } } I0507 06:28:27.817526 1928 raft_consensus.cc:366] T 9d26ac1dc0994a898fbb8aef9162ee61 P af7bb239a5d6454daa140682a17b532d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0507 06:28:27.817528 1929 raft_consensus.cc:366] T 9d26ac1dc0994a898fbb8aef9162ee61 P f9f586dd86b04fe984968b9de7f52209 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0507 06:28:27.817586 1929 raft_consensus.cc:705] T 9d26ac1dc0994a898fbb8aef9162ee61 P f9f586dd86b04fe984968b9de7f52209 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9f586dd86b04fe984968b9de7f52209, State: Initialized, Role: FOLLOWER I0507 06:28:27.817585 1928 raft_consensus.cc:705] T 9d26ac1dc0994a898fbb8aef9162ee61 P af7bb239a5d6454daa140682a17b532d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af7bb239a5d6454daa140682a17b532d, State: Initialized, Role: FOLLOWER I0507 06:28:27.818002 1929 consensus_queue.cc:229] T 9d26ac1dc0994a898fbb8aef9162ee61 P f9f586dd86b04fe984968b9de7f52209 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9f586dd86b04fe984968b9de7f52209" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 36235 } } peers { permanent_uuid: "af7bb239a5d6454daa140682a17b532d" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 43367 } } I0507 06:28:27.818003 1928 consensus_queue.cc:229] T 9d26ac1dc0994a898fbb8aef9162ee61 P af7bb239a5d6454daa140682a17b532d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9f586dd86b04fe984968b9de7f52209" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 36235 } } peers { permanent_uuid: "af7bb239a5d6454daa140682a17b532d" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 43367 } } I0507 06:28:27.818783 1911 heartbeater.cc:467] Master 127.0.57.254:38587 was elected leader, sending a full tablet report... I0507 06:28:27.819213 1929 ts_tablet_manager.cc:1081] T 9d26ac1dc0994a898fbb8aef9162ee61 P f9f586dd86b04fe984968b9de7f52209: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s I0507 06:28:27.819260 1928 ts_tablet_manager.cc:1081] T 9d26ac1dc0994a898fbb8aef9162ee61 P af7bb239a5d6454daa140682a17b532d: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s I0507 06:28:27.819382 1789 heartbeater.cc:467] Master 127.0.57.254:38587 was elected leader, sending a full tablet report... I0507 06:28:27.903564 1932 raft_consensus.cc:472] T 9d26ac1dc0994a898fbb8aef9162ee61 P af7bb239a5d6454daa140682a17b532d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0507 06:28:27.903802 1932 raft_consensus.cc:494] T 9d26ac1dc0994a898fbb8aef9162ee61 P af7bb239a5d6454daa140682a17b532d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9f586dd86b04fe984968b9de7f52209" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 36235 } } peers { permanent_uuid: "af7bb239a5d6454daa140682a17b532d" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 43367 } } I0507 06:28:27.905409 1932 leader_election.cc:296] T 9d26ac1dc0994a898fbb8aef9162ee61 P af7bb239a5d6454daa140682a17b532d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f9f586dd86b04fe984968b9de7f52209 (127.0.57.195:36235) I0507 06:28:27.910181 1868 tablet_service.cc:1286] Received RequestConsensusVote() RPC: tablet_id: "9d26ac1dc0994a898fbb8aef9162ee61" candidate_uuid: "af7bb239a5d6454daa140682a17b532d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9f586dd86b04fe984968b9de7f52209" is_pre_election: true I0507 06:28:27.910533 1868 raft_consensus.cc:2346] T 9d26ac1dc0994a898fbb8aef9162ee61 P f9f586dd86b04fe984968b9de7f52209 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af7bb239a5d6454daa140682a17b532d in term 0. I0507 06:28:27.911165 1679 leader_election.cc:310] T 9d26ac1dc0994a898fbb8aef9162ee61 P af7bb239a5d6454daa140682a17b532d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: af7bb239a5d6454daa140682a17b532d, f9f586dd86b04fe984968b9de7f52209; no voters: I0507 06:28:27.911648 1932 raft_consensus.cc:2640] T 9d26ac1dc0994a898fbb8aef9162ee61 P af7bb239a5d6454daa140682a17b532d [term 0 FOLLOWER]: Leader pre-election won for term 1 I0507 06:28:27.911728 1932 raft_consensus.cc:472] T 9d26ac1dc0994a898fbb8aef9162ee61 P af7bb239a5d6454daa140682a17b532d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0507 06:28:27.911811 1932 raft_consensus.cc:2886] T 9d26ac1dc0994a898fbb8aef9162ee61 P af7bb239a5d6454daa140682a17b532d [term 0 FOLLOWER]: Advancing to term 1 I0507 06:28:27.913589 1932 raft_consensus.cc:494] T 9d26ac1dc0994a898fbb8aef9162ee61 P af7bb239a5d6454daa140682a17b532d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9f586dd86b04fe984968b9de7f52209" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 36235 } } peers { permanent_uuid: "af7bb239a5d6454daa140682a17b532d" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 43367 } } I0507 06:28:27.913978 1932 leader_election.cc:296] T 9d26ac1dc0994a898fbb8aef9162ee61 P af7bb239a5d6454daa140682a17b532d [CANDIDATE]: Term 1 election: Requested vote from peers f9f586dd86b04fe984968b9de7f52209 (127.0.57.195:36235) I0507 06:28:27.914582 1868 tablet_service.cc:1286] Received RequestConsensusVote() RPC: tablet_id: "9d26ac1dc0994a898fbb8aef9162ee61" candidate_uuid: "af7bb239a5d6454daa140682a17b532d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9f586dd86b04fe984968b9de7f52209" I0507 06:28:27.914718 1868 raft_consensus.cc:2886] T 9d26ac1dc0994a898fbb8aef9162ee61 P f9f586dd86b04fe984968b9de7f52209 [term 0 FOLLOWER]: Advancing to term 1 I0507 06:28:27.916504 1868 raft_consensus.cc:2346] T 9d26ac1dc0994a898fbb8aef9162ee61 P f9f586dd86b04fe984968b9de7f52209 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af7bb239a5d6454daa140682a17b532d in term 1. I0507 06:28:27.917009 1679 leader_election.cc:310] T 9d26ac1dc0994a898fbb8aef9162ee61 P af7bb239a5d6454daa140682a17b532d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: af7bb239a5d6454daa140682a17b532d, f9f586dd86b04fe984968b9de7f52209; no voters: I0507 06:28:27.917292 1932 raft_consensus.cc:2640] T 9d26ac1dc0994a898fbb8aef9162ee61 P af7bb239a5d6454daa140682a17b532d [term 1 FOLLOWER]: Leader election won for term 1 I0507 06:28:27.917793 1932 raft_consensus.cc:667] T 9d26ac1dc0994a898fbb8aef9162ee61 P af7bb239a5d6454daa140682a17b532d [term 1 LEADER]: Becoming Leader. State: Replica: af7bb239a5d6454daa140682a17b532d, State: Running, Role: LEADER I0507 06:28:27.918150 1932 consensus_queue.cc:206] T 9d26ac1dc0994a898fbb8aef9162ee61 P af7bb239a5d6454daa140682a17b532d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9f586dd86b04fe984968b9de7f52209" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 36235 } } peers { permanent_uuid: "af7bb239a5d6454daa140682a17b532d" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 43367 } } I0507 06:28:27.922462 1501 catalog_manager.cc:3988] T 9d26ac1dc0994a898fbb8aef9162ee61 P af7bb239a5d6454daa140682a17b532d reported cstate change: term changed from 0 to 1, leader changed from to af7bb239a5d6454daa140682a17b532d (127.0.57.194). New cstate: current_term: 1 leader_uuid: "af7bb239a5d6454daa140682a17b532d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9f586dd86b04fe984968b9de7f52209" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 36235 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af7bb239a5d6454daa140682a17b532d" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 43367 } health_report { overall_health: HEALTHY } } } I0507 06:28:27.961305 231 external_mini_cluster.cc:623] 3 TS(s) registered with all masters I0507 06:28:27.963274 231 ts_itest-base.cc:232] Waiting for 0 tablets on tserver 180459af951d4aabbb5312ba1ae741bb to finish bootstrapping I0507 06:28:27.970710 231 ts_itest-base.cc:232] Waiting for 1 tablets on tserver af7bb239a5d6454daa140682a17b532d to finish bootstrapping I0507 06:28:27.976687 231 ts_itest-base.cc:232] Waiting for 1 tablets on tserver f9f586dd86b04fe984968b9de7f52209 to finish bootstrapping W0507 06:28:27.979622 1912 tablet.cc:1786] T 9d26ac1dc0994a898fbb8aef9162ee61 P f9f586dd86b04fe984968b9de7f52209: Can't schedule compaction. Clean time has not been advanced past its initial value. I0507 06:28:27.982290 231 test_util.cc:212] Using random seed: 1395340724 W0507 06:28:27.992514 1790 tablet.cc:1786] T 9d26ac1dc0994a898fbb8aef9162ee61 P af7bb239a5d6454daa140682a17b532d: Can't schedule compaction. Clean time has not been advanced past its initial value. I0507 06:28:27.993516 231 test_workload.cc:306] TestWorkload: Skipping table creation because table TestTable already exists I0507 06:28:28.038919 1868 raft_consensus.cc:1181] T 9d26ac1dc0994a898fbb8aef9162ee61 P f9f586dd86b04fe984968b9de7f52209 [term 1 FOLLOWER]: Refusing update from remote peer af7bb239a5d6454daa140682a17b532d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0507 06:28:28.039788 1932 consensus_queue.cc:984] T 9d26ac1dc0994a898fbb8aef9162ee61 P af7bb239a5d6454daa140682a17b532d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9f586dd86b04fe984968b9de7f52209" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 36235 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0507 06:28:28.053692 1948 mvcc.cc:203] Tried to move safe_time back from 6378334240917942272 to 6378334240435220480. Current Snapshot: MvccSnapshot[committed={T|T < 6378334240917942272}] I0507 06:28:28.058063 1947 mvcc.cc:203] Tried to move safe_time back from 6378334240917942272 to 6378334240435220480. Current Snapshot: MvccSnapshot[committed={T|T < 6378334240917942272}] I0507 06:28:28.108907 1746 consensus_queue.cc:206] T 9d26ac1dc0994a898fbb8aef9162ee61 P af7bb239a5d6454daa140682a17b532d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f9f586dd86b04fe984968b9de7f52209" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 36235 } } peers { permanent_uuid: "af7bb239a5d6454daa140682a17b532d" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 43367 } } peers { permanent_uuid: "180459af951d4aabbb5312ba1ae741bb" member_type: NON_VOTER last_known_addr { host: "127.0.57.193" port: 37035 } attrs { } } I0507 06:28:28.111155 1868 raft_consensus.cc:1181] T 9d26ac1dc0994a898fbb8aef9162ee61 P f9f586dd86b04fe984968b9de7f52209 [term 1 FOLLOWER]: Refusing update from remote peer af7bb239a5d6454daa140682a17b532d: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch) I0507 06:28:28.111702 1932 consensus_queue.cc:984] T 9d26ac1dc0994a898fbb8aef9162ee61 P af7bb239a5d6454daa140682a17b532d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9f586dd86b04fe984968b9de7f52209" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 36235 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s I0507 06:28:28.113901 1932 raft_consensus.cc:2788] T 9d26ac1dc0994a898fbb8aef9162ee61 P af7bb239a5d6454daa140682a17b532d [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index -1 to 10, NON_VOTER 180459af951d4aabbb5312ba1ae741bb (127.0.57.193) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f9f586dd86b04fe984968b9de7f52209" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 36235 } } peers { permanent_uuid: "af7bb239a5d6454daa140682a17b532d" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 43367 } } peers { permanent_uuid: "180459af951d4aabbb5312ba1ae741bb" member_type: NON_VOTER last_known_addr { host: "127.0.57.193" port: 37035 } attrs { } } } I0507 06:28:28.114351 1868 raft_consensus.cc:2788] T 9d26ac1dc0994a898fbb8aef9162ee61 P f9f586dd86b04fe984968b9de7f52209 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index -1 to 10, NON_VOTER 180459af951d4aabbb5312ba1ae741bb (127.0.57.193) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f9f586dd86b04fe984968b9de7f52209" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 36235 } } peers { permanent_uuid: "af7bb239a5d6454daa140682a17b532d" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 43367 } } peers { permanent_uuid: "180459af951d4aabbb5312ba1ae741bb" member_type: NON_VOTER last_known_addr { host: "127.0.57.193" port: 37035 } attrs { } } } W0507 06:28:28.116717 1679 consensus_peers.cc:457] T 9d26ac1dc0994a898fbb8aef9162ee61 P af7bb239a5d6454daa140682a17b532d -> Peer 180459af951d4aabbb5312ba1ae741bb (127.0.57.193:37035): Couldn't send request to peer 180459af951d4aabbb5312ba1ae741bb. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9d26ac1dc0994a898fbb8aef9162ee61. This is attempt 1: this message will repeat every 5th retry. I0507 06:28:28.118897 1495 catalog_manager.cc:3988] T 9d26ac1dc0994a898fbb8aef9162ee61 P af7bb239a5d6454daa140682a17b532d reported cstate change: config changed from index -1 to 10, NON_VOTER 180459af951d4aabbb5312ba1ae741bb (127.0.57.193) added. New cstate: current_term: 1 leader_uuid: "af7bb239a5d6454daa140682a17b532d" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f9f586dd86b04fe984968b9de7f52209" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 36235 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af7bb239a5d6454daa140682a17b532d" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 43367 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "180459af951d4aabbb5312ba1ae741bb" member_type: NON_VOTER last_known_addr { host: "127.0.57.193" port: 37035 } attrs { } health_report { overall_health: UNKNOWN } } } I0507 06:28:28.248209 1666 heartbeater.cc:467] Master 127.0.57.254:38587 was elected leader, sending a full tablet report... I0507 06:28:28.562016 1959 ts_tablet_manager.cc:676] T 9d26ac1dc0994a898fbb8aef9162ee61 P 180459af951d4aabbb5312ba1ae741bb: Initiating tablet copy from peer af7bb239a5d6454daa140682a17b532d (127.0.57.194:43367) I0507 06:28:28.562330 1959 tablet_copy_client.cc:240] T 9d26ac1dc0994a898fbb8aef9162ee61 P 180459af951d4aabbb5312ba1ae741bb: tablet copy: Beginning tablet copy session from remote peer at address 127.0.57.194:43367 I0507 06:28:28.567293 1766 tablet_copy_service.cc:135] P af7bb239a5d6454daa140682a17b532d: Received BeginTabletCopySession request for tablet 9d26ac1dc0994a898fbb8aef9162ee61 from peer 180459af951d4aabbb5312ba1ae741bb ({username='slave'} at 127.0.57.193:36941) I0507 06:28:28.567451 1766 tablet_copy_service.cc:156] P af7bb239a5d6454daa140682a17b532d: Beginning new tablet copy session on tablet 9d26ac1dc0994a898fbb8aef9162ee61 from peer 180459af951d4aabbb5312ba1ae741bb at {username='slave'} at 127.0.57.193:36941: session id = 180459af951d4aabbb5312ba1ae741bb-9d26ac1dc0994a898fbb8aef9162ee61 I0507 06:28:28.569727 1766 tablet_copy_source_session.cc:202] T 9d26ac1dc0994a898fbb8aef9162ee61 P af7bb239a5d6454daa140682a17b532d: Tablet Copy: opened 0 blocks and 1 log segments I0507 06:28:28.573237 1959 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet 9d26ac1dc0994a898fbb8aef9162ee61. 1 dirs total, 0 dirs full, 0 dirs failed I0507 06:28:28.582172 1959 tablet_copy_client.cc:575] T 9d26ac1dc0994a898fbb8aef9162ee61 P 180459af951d4aabbb5312ba1ae741bb: tablet copy: Starting download of 0 data blocks... I0507 06:28:28.582398 1959 tablet_copy_client.cc:538] T 9d26ac1dc0994a898fbb8aef9162ee61 P 180459af951d4aabbb5312ba1ae741bb: tablet copy: Starting download of 1 WAL segments... I0507 06:28:28.584208 1959 tablet_copy_client.cc:749] T 9d26ac1dc0994a898fbb8aef9162ee61 P 180459af951d4aabbb5312ba1ae741bb: tablet copy: Injecting latency into file download: 1000 I0507 06:28:29.584655 1959 tablet_copy_client.cc:414] T 9d26ac1dc0994a898fbb8aef9162ee61 P 180459af951d4aabbb5312ba1ae741bb: tablet copy: Tablet Copy complete. Replacing tablet superblock. I0507 06:28:29.587478 1959 tablet_bootstrap.cc:439] T 9d26ac1dc0994a898fbb8aef9162ee61 P 180459af951d4aabbb5312ba1ae741bb: Bootstrap starting. I0507 06:28:29.594341 1959 log.cc:525] T 9d26ac1dc0994a898fbb8aef9162ee61 P 180459af951d4aabbb5312ba1ae741bb: Log is configured to *not* fsync() on all Append() calls I0507 06:28:29.621986 1959 tablet_bootstrap.cc:439] T 9d26ac1dc0994a898fbb8aef9162ee61 P 180459af951d4aabbb5312ba1ae741bb: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=0} inserts{seen=400 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I0507 06:28:29.622396 1959 tablet_bootstrap.cc:439] T 9d26ac1dc0994a898fbb8aef9162ee61 P 180459af951d4aabbb5312ba1ae741bb: Bootstrap complete. I0507 06:28:29.622773 1959 ts_tablet_manager.cc:1053] T 9d26ac1dc0994a898fbb8aef9162ee61 P 180459af951d4aabbb5312ba1ae741bb: Time spent bootstrapping tablet: real 0.035s user 0.028s sys 0.008s I0507 06:28:29.627140 1959 raft_consensus.cc:340] T 9d26ac1dc0994a898fbb8aef9162ee61 P 180459af951d4aabbb5312ba1ae741bb [term 1 LEARNER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f9f586dd86b04fe984968b9de7f52209" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 36235 } } peers { permanent_uuid: "af7bb239a5d6454daa140682a17b532d" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 43367 } } peers { permanent_uuid: "180459af951d4aabbb5312ba1ae741bb" member_type: NON_VOTER last_known_addr { host: "127.0.57.193" port: 37035 } attrs { } } I0507 06:28:29.627480 1959 raft_consensus.cc:705] T 9d26ac1dc0994a898fbb8aef9162ee61 P 180459af951d4aabbb5312ba1ae741bb [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 180459af951d4aabbb5312ba1ae741bb, State: Initialized, Role: LEARNER I0507 06:28:29.627786 1959 consensus_queue.cc:229] T 9d26ac1dc0994a898fbb8aef9162ee61 P 180459af951d4aabbb5312ba1ae741bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 1.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f9f586dd86b04fe984968b9de7f52209" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 36235 } } peers { permanent_uuid: "af7bb239a5d6454daa140682a17b532d" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 43367 } } peers { permanent_uuid: "180459af951d4aabbb5312ba1ae741bb" member_type: NON_VOTER last_known_addr { host: "127.0.57.193" port: 37035 } attrs { } } I0507 06:28:29.628969 1959 ts_tablet_manager.cc:1081] T 9d26ac1dc0994a898fbb8aef9162ee61 P 180459af951d4aabbb5312ba1ae741bb: Time spent starting tablet: real 0.006s user 0.003s sys 0.005s I0507 06:28:29.629722 1766 tablet_copy_service.cc:325] P af7bb239a5d6454daa140682a17b532d: Request end of tablet copy session 180459af951d4aabbb5312ba1ae741bb-9d26ac1dc0994a898fbb8aef9162ee61 received from {username='slave'} at 127.0.57.193:36941 I0507 06:28:29.629830 1766 tablet_copy_service.cc:413] P af7bb239a5d6454daa140682a17b532d: ending tablet copy session 180459af951d4aabbb5312ba1ae741bb-9d26ac1dc0994a898fbb8aef9162ee61 on tablet 9d26ac1dc0994a898fbb8aef9162ee61 with peer 180459af951d4aabbb5312ba1ae741bb I0507 06:28:29.643949 1746 tablet_service.cc:1431] Received LeaderStepDown RPC: tablet_id: "9d26ac1dc0994a898fbb8aef9162ee61" dest_uuid: "af7bb239a5d6454daa140682a17b532d" from {username='slave'} at 127.0.0.1:52056 I0507 06:28:29.644145 1746 raft_consensus.cc:569] T 9d26ac1dc0994a898fbb8aef9162ee61 P af7bb239a5d6454daa140682a17b532d [term 1 LEADER]: Received request to step down I0507 06:28:29.644187 1746 raft_consensus.cc:2881] T 9d26ac1dc0994a898fbb8aef9162ee61 P af7bb239a5d6454daa140682a17b532d [term 1 LEADER]: Stepping down as leader of term 1 I0507 06:28:29.644225 1746 raft_consensus.cc:705] T 9d26ac1dc0994a898fbb8aef9162ee61 P af7bb239a5d6454daa140682a17b532d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: af7bb239a5d6454daa140682a17b532d, State: Running, Role: LEADER I0507 06:28:29.644425 1746 consensus_queue.cc:229] T 9d26ac1dc0994a898fbb8aef9162ee61 P af7bb239a5d6454daa140682a17b532d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 10, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f9f586dd86b04fe984968b9de7f52209" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 36235 } } peers { permanent_uuid: "af7bb239a5d6454daa140682a17b532d" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 43367 } } peers { permanent_uuid: "180459af951d4aabbb5312ba1ae741bb" member_type: NON_VOTER last_known_addr { host: "127.0.57.193" port: 37035 } attrs { } } I0507 06:28:29.644748 1746 raft_consensus.cc:2886] T 9d26ac1dc0994a898fbb8aef9162ee61 P af7bb239a5d6454daa140682a17b532d [term 1 FOLLOWER]: Advancing to term 2 W0507 06:28:29.656278 1706 tablet_service.cc:646] failed transaction from {username='slave'} at 127.0.0.1:52052: Illegal state: Replica af7bb239a5d6454daa140682a17b532d is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f9f586dd86b04fe984968b9de7f52209" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 36235 } } peers { permanent_uuid: "af7bb239a5d6454daa140682a17b532d" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 43367 } } peers { permanent_uuid: "180459af951d4aabbb5312ba1ae741bb" member_type: NON_VOTER last_known_addr { host: "127.0.57.193" port: 37035 } attrs { } } } W0507 06:28:29.656436 1704 tablet_service.cc:646] failed transaction from {username='slave'} at 127.0.0.1:52052: Illegal state: Replica af7bb239a5d6454daa140682a17b532d is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f9f586dd86b04fe984968b9de7f52209" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 36235 } } peers { permanent_uuid: "af7bb239a5d6454daa140682a17b532d" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 43367 } } peers { permanent_uuid: "180459af951d4aabbb5312ba1ae741bb" member_type: NON_VOTER last_known_addr { host: "127.0.57.193" port: 37035 } attrs { } } } W0507 06:28:29.656689 1705 tablet_service.cc:646] failed transaction from {username='slave'} at 127.0.0.1:52052: Illegal state: Replica af7bb239a5d6454daa140682a17b532d is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f9f586dd86b04fe984968b9de7f52209" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 36235 } } peers { permanent_uuid: "af7bb239a5d6454daa140682a17b532d" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 43367 } } peers { permanent_uuid: "180459af951d4aabbb5312ba1ae741bb" member_type: NON_VOTER last_known_addr { host: "127.0.57.193" port: 37035 } attrs { } } } W0507 06:28:29.657044 1703 tablet_service.cc:646] failed transaction from {username='slave'} at 127.0.0.1:52052: Illegal state: Replica af7bb239a5d6454daa140682a17b532d is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f9f586dd86b04fe984968b9de7f52209" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 36235 } } peers { permanent_uuid: "af7bb239a5d6454daa140682a17b532d" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 43367 } } peers { permanent_uuid: "180459af951d4aabbb5312ba1ae741bb" member_type: NON_VOTER last_known_addr { host: "127.0.57.193" port: 37035 } attrs { } } } W0507 06:28:29.668365 1828 tablet_service.cc:646] failed transaction from {username='slave'} at 127.0.0.1:58072: Illegal state: Replica f9f586dd86b04fe984968b9de7f52209 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 1 leader_uuid: "af7bb239a5d6454daa140682a17b532d" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f9f586dd86b04fe984968b9de7f52209" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 36235 } } peers { permanent_uuid: "af7bb239a5d6454daa140682a17b532d" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 43367 } } peers { permanent_uuid: "180459af951d4aabbb5312ba1ae741bb" member_type: NON_VOTER last_known_addr { host: "127.0.57.193" port: 37035 } attrs { } } } W0507 06:28:29.668686 1827 tablet_service.cc:646] failed transaction from {username='slave'} at 127.0.0.1:58072: Illegal state: Replica f9f586dd86b04fe984968b9de7f52209 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 1 leader_uuid: "af7bb239a5d6454daa140682a17b532d" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f9f586dd86b04fe984968b9de7f52209" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 36235 } } peers { permanent_uuid: "af7bb239a5d6454daa140682a17b532d" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 43367 } } peers { permanent_uuid: "180459af951d4aabbb5312ba1ae741bb" member_type: NON_VOTER last_known_addr { host: "127.0.57.193" port: 37035 } attrs { } } } W0507 06:28:29.668998 1825 tablet_service.cc:646] failed transaction from {username='slave'} at 127.0.0.1:58072: Illegal state: Replica f9f586dd86b04fe984968b9de7f52209 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 1 leader_uuid: "af7bb239a5d6454daa140682a17b532d" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f9f586dd86b04fe984968b9de7f52209" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 36235 } } peers { permanent_uuid: "af7bb239a5d6454daa140682a17b532d" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 43367 } } peers { permanent_uuid: "180459af951d4aabbb5312ba1ae741bb" member_type: NON_VOTER last_known_addr { host: "127.0.57.193" port: 37035 } attrs { } } } W0507 06:28:29.669463 1826 tablet_service.cc:646] failed transaction from {username='slave'} at 127.0.0.1:58072: Illegal state: Replica f9f586dd86b04fe984968b9de7f52209 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 1 leader_uuid: "af7bb239a5d6454daa140682a17b532d" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f9f586dd86b04fe984968b9de7f52209" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 36235 } } peers { permanent_uuid: "af7bb239a5d6454daa140682a17b532d" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 43367 } } peers { permanent_uuid: "180459af951d4aabbb5312ba1ae741bb" member_type: NON_VOTER last_known_addr { host: "127.0.57.193" port: 37035 } attrs { } } } W0507 06:28:29.674746 1703 tablet_service.cc:646] failed transaction from {username='slave'} at 127.0.0.1:52052: Illegal state: Replica af7bb239a5d6454daa140682a17b532d is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f9f586dd86b04fe984968b9de7f52209" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 36235 } } peers { permanent_uuid: "af7bb239a5d6454daa140682a17b532d" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 43367 } } peers { permanent_uuid: "180459af951d4aabbb5312ba1ae741bb" member_type: NON_VOTER last_known_addr { host: "127.0.57.193" port: 37035 } attrs { } } } W0507 06:28:29.676017 1706 tablet_service.cc:646] failed transaction from {username='slave'} at 127.0.0.1:52052: Illegal state: Replica af7bb239a5d6454daa140682a17b532d is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f9f586dd86b04fe984968b9de7f52209" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 36235 } } peers { permanent_uuid: "af7bb239a5d6454daa140682a17b532d" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 43367 } } peers { permanent_uuid: "180459af951d4aabbb5312ba1ae741bb" member_type: NON_VOTER last_known_addr { host: "127.0.57.193" port: 37035 } attrs { } } } W0507 06:28:29.676245 1703 tablet_service.cc:646] failed transaction from {username='slave'} at 127.0.0.1:52052: Illegal state: Replica af7bb239a5d6454daa140682a17b532d is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f9f586dd86b04fe984968b9de7f52209" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 36235 } } peers { permanent_uuid: "af7bb239a5d6454daa140682a17b532d" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 43367 } } peers { permanent_uuid: "180459af951d4aabbb5312ba1ae741bb" member_type: NON_VOTER last_known_addr { host: "127.0.57.193" port: 37035 } attrs { } } } W0507 06:28:29.677651 1703 tablet_service.cc:646] failed transaction from {username='slave'} at 127.0.0.1:52052: Illegal state: Replica af7bb239a5d6454daa140682a17b532d is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f9f586dd86b04fe984968b9de7f52209" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 36235 } } peers { permanent_uuid: "af7bb239a5d6454daa140682a17b532d" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 43367 } } peers { permanent_uuid: "180459af951d4aabbb5312ba1ae741bb" member_type: NON_VOTER last_known_addr { host: "127.0.57.193" port: 37035 } attrs { } } } W0507 06:28:29.680133 1826 tablet_service.cc:646] failed transaction from {username='slave'} at 127.0.0.1:58072: Illegal state: Replica f9f586dd86b04fe984968b9de7f52209 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 1 leader_uuid: "af7bb239a5d6454daa140682a17b532d" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f9f586dd86b04fe984968b9de7f52209" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 36235 } } peers { permanent_uuid: "af7bb239a5d6454daa140682a17b532d" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 43367 } } peers { permanent_uuid: "180459af951d4aabbb5312ba1ae741bb" member_type: NON_VOTER last_known_addr { host: "127.0.57.193" port: 37035 } attrs { } } } W0507 06:28:29.684568 1826 tablet_service.cc:646] failed transaction from {username='slave'} at 127.0.0.1:58072: Illegal state: Replica f9f586dd86b04fe984968b9de7f52209 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 1 leader_uuid: "af7bb239a5d6454daa140682a17b532d" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f9f586dd86b04fe984968b9de7f52209" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 36235 } } peers { permanent_uuid: "af7bb239a5d6454daa140682a17b532d" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 43367 } } peers { permanent_uuid: "180459af951d4aabbb5312ba1ae741bb" member_type: NON_VOTER last_known_addr { host: "127.0.57.193" port: 37035 } attrs { } } } W0507 06:28:29.684736 1827 tablet_service.cc:646] failed transaction from {username='slave'} at 127.0.0.1:58072: Illegal state: Replica f9f586dd86b04fe984968b9de7f52209 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 1 leader_uuid: "af7bb239a5d6454daa140682a17b532d" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f9f586dd86b04fe984968b9de7f52209" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 36235 } } peers { permanent_uuid: "af7bb239a5d6454daa140682a17b532d" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 43367 } } peers { permanent_uuid: "180459af951d4aabbb5312ba1ae741bb" member_type: NON_VOTER last_known_addr { host: "127.0.57.193" port: 37035 } attrs { } } } W0507 06:28:29.684998 1828 tablet_service.cc:646] failed transaction from {username='slave'} at 127.0.0.1:58072: Illegal state: Replica f9f586dd86b04fe984968b9de7f52209 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 1 leader_uuid: "af7bb239a5d6454daa140682a17b532d" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f9f586dd86b04fe984968b9de7f52209" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 36235 } } peers { permanent_uuid: "af7bb239a5d6454daa140682a17b532d" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 43367 } } peers { permanent_uuid: "180459af951d4aabbb5312ba1ae741bb" member_type: NON_VOTER last_known_addr { host: "127.0.57.193" port: 37035 } attrs { } } } W0507 06:28:29.687031 1703 tablet_service.cc:646] failed transaction from {username='slave'} at 127.0.0.1:52052: Illegal state: Replica af7bb239a5d6454daa140682a17b532d is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f9f586dd86b04fe984968b9de7f52209" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 36235 } } peers { permanent_uuid: "af7bb239a5d6454daa140682a17b532d" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 43367 } } peers { permanent_uuid: "180459af951d4aabbb5312ba1ae741bb" member_type: NON_VOTER last_known_addr { host: "127.0.57.193" port: 37035 } attrs { } } } W0507 06:28:29.693717 1703 tablet_service.cc:646] failed transaction from {username='slave'} at 127.0.0.1:52052: Illegal state: Replica af7bb239a5d6454daa140682a17b532d is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f9f586dd86b04fe984968b9de7f52209" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 36235 } } peers { permanent_uuid: "af7bb239a5d6454daa140682a17b532d" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 43367 } } peers { permanent_uuid: "180459af951d4aabbb5312ba1ae741bb" member_type: NON_VOTER last_known_addr { host: "127.0.57.193" port: 37035 } attrs { } } } W0507 06:28:29.695555 1703 tablet_service.cc:646] failed transaction from {username='slave'} at 127.0.0.1:52052: Illegal state: Replica af7bb239a5d6454daa140682a17b532d is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f9f586dd86b04fe984968b9de7f52209" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 36235 } } peers { permanent_uuid: "af7bb239a5d6454daa140682a17b532d" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 43367 } } peers { permanent_uuid: "180459af951d4aabbb5312ba1ae741bb" member_type: NON_VOTER last_known_addr { host: "127.0.57.193" port: 37035 } attrs { } } } W0507 06:28:29.695776 1828 tablet_service.cc:646] failed transaction from {username='slave'} at 127.0.0.1:58072: Illegal state: Replica f9f586dd86b04fe984968b9de7f52209 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 1 leader_uuid: "af7bb239a5d6454daa140682a17b532d" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f9f586dd86b04fe984968b9de7f52209" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 36235 } } peers { permanent_uuid: "af7bb239a5d6454daa140682a17b532d" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 43367 } } peers { permanent_uuid: "180459af951d4aabbb5312ba1ae741bb" member_type: NON_VOTER last_known_addr { host: "127.0.57.193" port: 37035 } attrs { } } } W0507 06:28:29.696079 1706 tablet_service.cc:646] failed transaction from {username='slave'} at 127.0.0.1:52052: Illegal state: Replica af7bb239a5d6454daa140682a17b532d is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f9f586dd86b04fe984968b9de7f52209" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 36235 } } peers { permanent_uuid: "af7bb239a5d6454daa140682a17b532d" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 43367 } } peers { permanent_uuid: "180459af951d4aabbb5312ba1ae741bb" member_type: NON_VOTER last_known_addr { host: "127.0.57.193" port: 37035 } attrs { } } } W0507 06:28:29.702544 1828 tablet_service.cc:646] failed transaction from {username='slave'} at 127.0.0.1:58072: Illegal state: Replica f9f586dd86b04fe984968b9de7f52209 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 1 leader_uuid: "af7bb239a5d6454daa140682a17b532d" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f9f586dd86b04fe984968b9de7f52209" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 36235 } } peers { permanent_uuid: "af7bb239a5d6454daa140682a17b532d" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 43367 } } peers { permanent_uuid: "180459af951d4aabbb5312ba1ae741bb" member_type: NON_VOTER last_known_addr { host: "127.0.57.193" port: 37035 } attrs { } } } W0507 06:28:29.703682 1828 tablet_service.cc:646] failed transaction from {username='slave'} at 127.0.0.1:58072: Illegal state: Replica f9f586dd86b04fe984968b9de7f52209 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 1 leader_uuid: "af7bb239a5d6454daa140682a17b532d" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f9f586dd86b04fe984968b9de7f52209" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 36235 } } peers { permanent_uuid: "af7bb239a5d6454daa140682a17b532d" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 43367 } } peers { permanent_uuid: "180459af951d4aabbb5312ba1ae741bb" member_type: NON_VOTER last_known_addr { host: "127.0.57.193" port: 37035 } attrs { } } } W0507 06:28:29.706929 1828 tablet_service.cc:646] failed transaction from {username='slave'} at 127.0.0.1:58072: Illegal state: Replica f9f586dd86b04fe984968b9de7f52209 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 1 leader_uuid: "af7bb239a5d6454daa140682a17b532d" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f9f586dd86b04fe984968b9de7f52209" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 36235 } } peers { permanent_uuid: "af7bb239a5d6454daa140682a17b532d" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 43367 } } peers { permanent_uuid: "180459af951d4aabbb5312ba1ae741bb" member_type: NON_VOTER last_known_addr { host: "127.0.57.193" port: 37035 } attrs { } } } W0507 06:28:29.707298 1706 tablet_service.cc:646] failed transaction from {username='slave'} at 127.0.0.1:52052: Illegal state: Replica af7bb239a5d6454daa140682a17b532d is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f9f586dd86b04fe984968b9de7f52209" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 36235 } } peers { permanent_uuid: "af7bb239a5d6454daa140682a17b532d" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 43367 } } peers { permanent_uuid: "180459af951d4aabbb5312ba1ae741bb" member_type: NON_VOTER last_known_addr { host: "127.0.57.193" port: 37035 } attrs { } } } W0507 06:28:29.711525 1706 tablet_service.cc:646] failed transaction from {username='slave'} at 127.0.0.1:52052: Illegal state: Replica af7bb239a5d6454daa140682a17b532d is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f9f586dd86b04fe984968b9de7f52209" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 36235 } } peers { permanent_uuid: "af7bb239a5d6454daa140682a17b532d" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 43367 } } peers { permanent_uuid: "180459af951d4aabbb5312ba1ae741bb" member_type: NON_VOTER last_known_addr { host: "127.0.57.193" port: 37035 } attrs { } } } W0507 06:28:29.714603 1706 tablet_service.cc:646] failed transaction from {username='slave'} at 127.0.0.1:52052: Illegal state: Replica af7bb239a5d6454daa140682a17b532d is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f9f586dd86b04fe984968b9de7f52209" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 36235 } } peers { permanent_uuid: "af7bb239a5d6454daa140682a17b532d" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 43367 } } peers { permanent_uuid: "180459af951d4aabbb5312ba1ae741bb" member_type: NON_VOTER last_known_addr { host: "127.0.57.193" port: 37035 } attrs { } } } W0507 06:28:29.718940 1706 tablet_service.cc:646] failed transaction from {username='slave'} at 127.0.0.1:52052: Illegal state: Replica af7bb239a5d6454daa140682a17b532d is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f9f586dd86b04fe984968b9de7f52209" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 36235 } } peers { permanent_uuid: "af7bb239a5d6454daa140682a17b532d" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 43367 } } peers { permanent_uuid: "180459af951d4aabbb5312ba1ae741bb" member_type: NON_VOTER last_known_addr { host: "127.0.57.193" port: 37035 } attrs { } } } W0507 06:28:29.720259 1828 tablet_service.cc:646] failed transaction from {username='slave'} at 127.0.0.1:58072: Illegal state: Replica f9f586dd86b04fe984968b9de7f52209 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 1 leader_uuid: "af7bb239a5d6454daa140682a17b532d" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f9f586dd86b04fe984968b9de7f52209" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 36235 } } peers { permanent_uuid: "af7bb239a5d6454daa140682a17b532d" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 43367 } } peers { permanent_uuid: "180459af951d4aabbb5312ba1ae741bb" member_type: NON_VOTER last_known_addr { host: "127.0.57.193" port: 37035 } attrs { } } } W0507 06:28:29.723681 1828 tablet_service.cc:646] failed transaction from {username='slave'} at 127.0.0.1:58072: Illegal state: Replica f9f586dd86b04fe984968b9de7f52209 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 1 leader_uuid: "af7bb239a5d6454daa140682a17b532d" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f9f586dd86b04fe984968b9de7f52209" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 36235 } } peers { permanent_uuid: "af7bb239a5d6454daa140682a17b532d" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 43367 } } peers { permanent_uuid: "180459af951d4aabbb5312ba1ae741bb" member_type: NON_VOTER last_known_addr { host: "127.0.57.193" port: 37035 } attrs { } } } W0507 06:28:29.724815 1828 tablet_service.cc:646] failed transaction from {username='slave'} at 127.0.0.1:58072: Illegal state: Replica f9f586dd86b04fe984968b9de7f52209 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 1 leader_uuid: "af7bb239a5d6454daa140682a17b532d" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f9f586dd86b04fe984968b9de7f52209" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 36235 } } peers { permanent_uuid: "af7bb239a5d6454daa140682a17b532d" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 43367 } } peers { permanent_uuid: "180459af951d4aabbb5312ba1ae741bb" member_type: NON_VOTER last_known_addr { host: "127.0.57.193" port: 37035 } attrs { } } } W0507 06:28:29.728098 1828 tablet_service.cc:646] failed transaction from {username='slave'} at 127.0.0.1:58072: Illegal state: Replica f9f586dd86b04fe984968b9de7f52209 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 1 leader_uuid: "af7bb239a5d6454daa140682a17b532d" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f9f586dd86b04fe984968b9de7f52209" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 36235 } } peers { permanent_uuid: "af7bb239a5d6454daa140682a17b532d" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 43367 } } peers { permanent_uuid: "180459af951d4aabbb5312ba1ae741bb" member_type: NON_VOTER last_known_addr { host: "127.0.57.193" port: 37035 } attrs { } } } W0507 06:28:29.735271 1706 tablet_service.cc:646] failed transaction from {username='slave'} at 127.0.0.1:52052: Illegal state: Replica af7bb239a5d6454daa140682a17b532d is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f9f586dd86b04fe984968b9de7f52209" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 36235 } } peers { permanent_uuid: "af7bb239a5d6454daa140682a17b532d" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 43367 } } peers { permanent_uuid: "180459af951d4aabbb5312ba1ae741bb" member_type: NON_VOTER last_known_addr { host: "127.0.57.193" port: 37035 } attrs { } } } W0507 06:28:29.735491 1703 tablet_service.cc:646] failed transaction from {username='slave'} at 127.0.0.1:52052: Illegal state: Replica af7bb239a5d6454daa140682a17b532d is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f9f586dd86b04fe984968b9de7f52209" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 36235 } } peers { permanent_uuid: "af7bb239a5d6454daa140682a17b532d" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 43367 } } peers { permanent_uuid: "180459af951d4aabbb5312ba1ae741bb" member_type: NON_VOTER last_known_addr { host: "127.0.57.193" port: 37035 } attrs { } } } W0507 06:28:29.740070 1703 tablet_service.cc:646] failed transaction from {username='slave'} at 127.0.0.1:52052: Illegal state: Replica af7bb239a5d6454daa140682a17b532d is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f9f586dd86b04fe984968b9de7f52209" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 36235 } } peers { permanent_uuid: "af7bb239a5d6454daa140682a17b532d" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 43367 } } peers { permanent_uuid: "180459af951d4aabbb5312ba1ae741bb" member_type: NON_VOTER last_known_addr { host: "127.0.57.193" port: 37035 } attrs { } } } W0507 06:28:29.741205 1703 tablet_service.cc:646] failed transaction from {username='slave'} at 127.0.0.1:52052: Illegal state: Replica af7bb239a5d6454daa140682a17b532d is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f9f586dd86b04fe984968b9de7f52209" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 36235 } } peers { permanent_uuid: "af7bb239a5d6454daa140682a17b532d" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 43367 } } peers { permanent_uuid: "180459af951d4aabbb5312ba1ae741bb" member_type: NON_VOTER last_known_addr { host: "127.0.57.193" port: 37035 } attrs { } } } W0507 06:28:29.747928 1828 tablet_service.cc:646] failed transaction from {username='slave'} at 127.0.0.1:58072: Illegal state: Replica f9f586dd86b04fe984968b9de7f52209 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 1 leader_uuid: "af7bb239a5d6454daa140682a17b532d" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f9f586dd86b04fe984968b9de7f52209" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 36235 } } peers { permanent_uuid: "af7bb239a5d6454daa140682a17b532d" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 43367 } } peers { permanent_uuid: "180459af951d4aabbb5312ba1ae741bb" member_type: NON_VOTER last_known_addr { host: "127.0.57.193" port: 37035 } attrs { } } } W0507 06:28:29.749071 1828 tablet_service.cc:646] failed transaction from {username='slave'} at 127.0.0.1:58072: Illegal state: Replica f9f586dd86b04fe984968b9de7f52209 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 1 leader_uuid: "af7bb239a5d6454daa140682a17b532d" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f9f586dd86b04fe984968b9de7f52209" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 36235 } } peers { permanent_uuid: "af7bb239a5d6454daa140682a17b532d" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 43367 } } peers { permanent_uuid: "180459af951d4aabbb5312ba1ae741bb" member_type: NON_VOTER last_known_addr { host: "127.0.57.193" port: 37035 } attrs { } } } W0507 06:28:29.752426 1828 tablet_service.cc:646] failed transaction from {username='slave'} at 127.0.0.1:58072: Illegal state: Replica f9f586dd86b04fe984968b9de7f52209 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 1 leader_uuid: "af7bb239a5d6454daa140682a17b532d" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f9f586dd86b04fe984968b9de7f52209" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 36235 } } peers { permanent_uuid: "af7bb239a5d6454daa140682a17b532d" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 43367 } } peers { permanent_uuid: "180459af951d4aabbb5312ba1ae741bb" member_type: NON_VOTER last_known_addr { host: "127.0.57.193" port: 37035 } attrs { } } } W0507 06:28:29.752673 1827 tablet_service.cc:646] failed transaction from {username='slave'} at 127.0.0.1:58072: Illegal state: Replica f9f586dd86b04fe984968b9de7f52209 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 1 leader_uuid: "af7bb239a5d6454daa140682a17b532d" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f9f586dd86b04fe984968b9de7f52209" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 36235 } } peers { permanent_uuid: "af7bb239a5d6454daa140682a17b532d" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 43367 } } peers { permanent_uuid: "180459af951d4aabbb5312ba1ae741bb" member_type: NON_VOTER last_known_addr { host: "127.0.57.193" port: 37035 } attrs { } } } W0507 06:28:29.765357 1703 tablet_service.cc:646] failed transaction from {username='slave'} at 127.0.0.1:52052: Illegal state: Replica af7bb239a5d6454daa140682a17b532d is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f9f586dd86b04fe984968b9de7f52209" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 36235 } } peers { permanent_uuid: "af7bb239a5d6454daa140682a17b532d" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 43367 } } peers { permanent_uuid: "180459af951d4aabbb5312ba1ae741bb" member_type: NON_VOTER last_known_addr { host: "127.0.57.193" port: 37035 } attrs { } } } W0507 06:28:29.766564 1703 tablet_service.cc:646] failed transaction from {username='slave'} at 127.0.0.1:52052: Illegal state: Replica af7bb239a5d6454daa140682a17b532d is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f9f586dd86b04fe984968b9de7f52209" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 36235 } } peers { permanent_uuid: "af7bb239a5d6454daa140682a17b532d" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 43367 } } peers { permanent_uuid: "180459af951d4aabbb5312ba1ae741bb" member_type: NON_VOTER last_known_addr { host: "127.0.57.193" port: 37035 } attrs { } } } W0507 06:28:29.766808 1706 tablet_service.cc:646] failed transaction from {username='slave'} at 127.0.0.1:52052: Illegal state: Replica af7bb239a5d6454daa140682a17b532d is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f9f586dd86b04fe984968b9de7f52209" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 36235 } } peers { permanent_uuid: "af7bb239a5d6454daa140682a17b532d" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 43367 } } peers { permanent_uuid: "180459af951d4aabbb5312ba1ae741bb" member_type: NON_VOTER last_known_addr { host: "127.0.57.193" port: 37035 } attrs { } } } W0507 06:28:29.770018 1706 tablet_service.cc:646] failed transaction from {username='slave'} at 127.0.0.1:52052: Illegal state: Replica af7bb239a5d6454daa140682a17b532d is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f9f586dd86b04fe984968b9de7f52209" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 36235 } } peers { permanent_uuid: "af7bb239a5d6454daa140682a17b532d" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 43367 } } peers { permanent_uuid: "180459af951d4aabbb5312ba1ae741bb" member_type: NON_VOTER last_known_addr { host: "127.0.57.193" port: 37035 } attrs { } } } W0507 06:28:29.779067 1827 tablet_service.cc:646] failed transaction from {username='slave'} at 127.0.0.1:58072: Illegal state: Replica f9f586dd86b04fe984968b9de7f52209 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 1 leader_uuid: "af7bb239a5d6454daa140682a17b532d" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f9f586dd86b04fe984968b9de7f52209" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 36235 } } peers { permanent_uuid: "af7bb239a5d6454daa140682a17b532d" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 43367 } } peers { permanent_uuid: "180459af951d4aabbb5312ba1ae741bb" member_type: NON_VOTER last_known_addr { host: "127.0.57.193" port: 37035 } attrs { } } } W0507 06:28:29.780019 1827 tablet_service.cc:646] failed transaction from {username='slave'} at 127.0.0.1:58072: Illegal state: Replica f9f586dd86b04fe984968b9de7f52209 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 1 leader_uuid: "af7bb239a5d6454daa140682a17b532d" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f9f586dd86b04fe984968b9de7f52209" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 36235 } } peers { permanent_uuid: "af7bb239a5d6454daa140682a17b532d" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 43367 } } peers { permanent_uuid: "180459af951d4aabbb5312ba1ae741bb" member_type: NON_VOTER last_known_addr { host: "127.0.57.193" port: 37035 } attrs { } } } W0507 06:28:29.781066 1827 tablet_service.cc:646] failed transaction from {username='slave'} at 127.0.0.1:58072: Illegal state: Replica f9f586dd86b04fe984968b9de7f52209 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 1 leader_uuid: "af7bb239a5d6454daa140682a17b532d" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f9f586dd86b04fe984968b9de7f52209" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 36235 } } peers { permanent_uuid: "af7bb239a5d6454daa140682a17b532d" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 43367 } } peers { permanent_uuid: "180459af951d4aabbb5312ba1ae741bb" member_type: NON_VOTER last_known_addr { host: "127.0.57.193" port: 37035 } attrs { } } } W0507 06:28:29.785634 1827 tablet_service.cc:646] failed transaction from {username='slave'} at 127.0.0.1:58072: Illegal state: Replica f9f586dd86b04fe984968b9de7f52209 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 1 leader_uuid: "af7bb239a5d6454daa140682a17b532d" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f9f586dd86b04fe984968b9de7f52209" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 36235 } } peers { permanent_uuid: "af7bb239a5d6454daa140682a17b532d" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 43367 } } peers { permanent_uuid: "180459af951d4aabbb5312ba1ae741bb" member_type: NON_VOTER last_known_addr { host: "127.0.57.193" port: 37035 } attrs { } } } W0507 06:28:29.797307 1827 tablet_service.cc:646] failed transaction from {username='slave'} at 127.0.0.1:58072: Illegal state: Replica f9f586dd86b04fe984968b9de7f52209 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 1 leader_uuid: "af7bb239a5d6454daa140682a17b532d" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f9f586dd86b04fe984968b9de7f52209" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 36235 } } peers { permanent_uuid: "af7bb239a5d6454daa140682a17b532d" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 43367 } } peers { permanent_uuid: "180459af951d4aabbb5312ba1ae741bb" member_type: NON_VOTER last_known_addr { host: "127.0.57.193" port: 37035 } attrs { } } } W0507 06:28:29.797505 1828 tablet_service.cc:646] failed transaction from {username='slave'} at 127.0.0.1:58072: Illegal state: Replica f9f586dd86b04fe984968b9de7f52209 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 1 leader_uuid: "af7bb239a5d6454daa140682a17b532d" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f9f586dd86b04fe984968b9de7f52209" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 36235 } } peers { permanent_uuid: "af7bb239a5d6454daa140682a17b532d" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 43367 } } peers { permanent_uuid: "180459af951d4aabbb5312ba1ae741bb" member_type: NON_VOTER last_known_addr { host: "127.0.57.193" port: 37035 } attrs { } } } W0507 06:28:29.798328 1706 tablet_service.cc:646] failed transaction from {username='slave'} at 127.0.0.1:52052: Illegal state: Replica af7bb239a5d6454daa140682a17b532d is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f9f586dd86b04fe984968b9de7f52209" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 36235 } } peers { permanent_uuid: "af7bb239a5d6454daa140682a17b532d" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 43367 } } peers { permanent_uuid: "180459af951d4aabbb5312ba1ae741bb" member_type: NON_VOTER last_known_addr { host: "127.0.57.193" port: 37035 } attrs { } } } W0507 06:28:29.800824 1706 tablet_service.cc:646] failed transaction from {username='slave'} at 127.0.0.1:52052: Illegal state: Replica af7bb239a5d6454daa140682a17b532d is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f9f586dd86b04fe984968b9de7f52209" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 36235 } } peers { permanent_uuid: "af7bb239a5d6454daa140682a17b532d" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 43367 } } peers { permanent_uuid: "180459af951d4aabbb5312ba1ae741bb" member_type: NON_VOTER last_known_addr { host: "127.0.57.193" port: 37035 } attrs { } } } W0507 06:28:29.812389 1706 tablet_service.cc:646] failed transaction from {username='slave'} at 127.0.0.1:52052: Illegal state: Replica af7bb239a5d6454daa140682a17b532d is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f9f586dd86b04fe984968b9de7f52209" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 36235 } } peers { permanent_uuid: "af7bb239a5d6454daa140682a17b532d" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 43367 } } peers { permanent_uuid: "180459af951d4aabbb5312ba1ae741bb" member_type: NON_VOTER last_known_addr { host: "127.0.57.193" port: 37035 } attrs { } } } W0507 06:28:29.813524 1706 tablet_service.cc:646] failed transaction from {username='slave'} at 127.0.0.1:52052: Illegal state: Replica af7bb239a5d6454daa140682a17b532d is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f9f586dd86b04fe984968b9de7f52209" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 36235 } } peers { permanent_uuid: "af7bb239a5d6454daa140682a17b532d" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 43367 } } peers { permanent_uuid: "180459af951d4aabbb5312ba1ae741bb" member_type: NON_VOTER last_known_addr { host: "127.0.57.193" port: 37035 } attrs { } } } W0507 06:28:29.817363 1827 tablet_service.cc:646] failed transaction from {username='slave'} at 127.0.0.1:58072: Illegal state: Replica f9f586dd86b04fe984968b9de7f52209 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 1 leader_uuid: "af7bb239a5d6454daa140682a17b532d" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f9f586dd86b04fe984968b9de7f52209" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 36235 } } peers { permanent_uuid: "af7bb239a5d6454daa140682a17b532d" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 43367 } } peers { permanent_uuid: "180459af951d4aabbb5312ba1ae741bb" member_type: NON_VOTER last_known_addr { host: "127.0.57.193" port: 37035 } attrs { } } } W0507 06:28:29.817595 1828 tablet_service.cc:646] failed transaction from {username='slave'} at 127.0.0.1:58072: Illegal state: Replica f9f586dd86b04fe984968b9de7f52209 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 1 leader_uuid: "af7bb239a5d6454daa140682a17b532d" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f9f586dd86b04fe984968b9de7f52209" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 36235 } } peers { permanent_uuid: "af7bb239a5d6454daa140682a17b532d" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 43367 } } peers { permanent_uuid: "180459af951d4aabbb5312ba1ae741bb" member_type: NON_VOTER last_known_addr { host: "127.0.57.193" port: 37035 } attrs { } } } W0507 06:28:29.831372 1706 tablet_service.cc:646] failed transaction from {username='slave'} at 127.0.0.1:52052: Illegal state: Replica af7bb239a5d6454daa140682a17b532d is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f9f586dd86b04fe984968b9de7f52209" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 36235 } } peers { permanent_uuid: "af7bb239a5d6454daa140682a17b532d" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 43367 } } peers { permanent_uuid: "180459af951d4aabbb5312ba1ae741bb" member_type: NON_VOTER last_known_addr { host: "127.0.57.193" port: 37035 } attrs { } } } W0507 06:28:29.832309 1703 tablet_service.cc:646] failed transaction from {username='slave'} at 127.0.0.1:52052: Illegal state: Replica af7bb239a5d6454daa140682a17b532d is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f9f586dd86b04fe984968b9de7f52209" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 36235 } } peers { permanent_uuid: "af7bb239a5d6454daa140682a17b532d" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 43367 } } peers { permanent_uuid: "180459af951d4aabbb5312ba1ae741bb" member_type: NON_VOTER last_known_addr { host: "127.0.57.193" port: 37035 } attrs { } } } W0507 06:28:29.836079 1703 tablet_service.cc:646] failed transaction from {username='slave'} at 127.0.0.1:52052: Illegal state: Replica af7bb239a5d6454daa140682a17b532d is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f9f586dd86b04fe984968b9de7f52209" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 36235 } } peers { permanent_uuid: "af7bb239a5d6454daa140682a17b532d" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 43367 } } peers { permanent_uuid: "180459af951d4aabbb5312ba1ae741bb" member_type: NON_VOTER last_known_addr { host: "127.0.57.193" port: 37035 } attrs { } } } W0507 06:28:29.838308 1703 tablet_service.cc:646] failed transaction from {username='slave'} at 127.0.0.1:52052: Illegal state: Replica af7bb239a5d6454daa140682a17b532d is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f9f586dd86b04fe984968b9de7f52209" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 36235 } } peers { permanent_uuid: "af7bb239a5d6454daa140682a17b532d" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 43367 } } peers { permanent_uuid: "180459af951d4aabbb5312ba1ae741bb" member_type: NON_VOTER last_known_addr { host: "127.0.57.193" port: 37035 } attrs { } } } W0507 06:28:29.850112 1828 tablet_service.cc:646] failed transaction from {username='slave'} at 127.0.0.1:58072: Illegal state: Replica f9f586dd86b04fe984968b9de7f52209 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 1 leader_uuid: "af7bb239a5d6454daa140682a17b532d" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f9f586dd86b04fe984968b9de7f52209" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 36235 } } peers { permanent_uuid: "af7bb239a5d6454daa140682a17b532d" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 43367 } } peers { permanent_uuid: "180459af951d4aabbb5312ba1ae741bb" member_type: NON_VOTER last_known_addr { host: "127.0.57.193" port: 37035 } attrs { } } } W0507 06:28:29.851047 1828 tablet_service.cc:646] failed transaction from {username='slave'} at 127.0.0.1:58072: Illegal state: Replica f9f586dd86b04fe984968b9de7f52209 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 1 leader_uuid: "af7bb239a5d6454daa140682a17b532d" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f9f586dd86b04fe984968b9de7f52209" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 36235 } } peers { permanent_uuid: "af7bb239a5d6454daa140682a17b532d" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 43367 } } peers { permanent_uuid: "180459af951d4aabbb5312ba1ae741bb" member_type: NON_VOTER last_known_addr { host: "127.0.57.193" port: 37035 } attrs { } } } W0507 06:28:29.856308 1828 tablet_service.cc:646] failed transaction from {username='slave'} at 127.0.0.1:58072: Illegal state: Replica f9f586dd86b04fe984968b9de7f52209 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 1 leader_uuid: "af7bb239a5d6454daa140682a17b532d" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f9f586dd86b04fe984968b9de7f52209" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 36235 } } peers { permanent_uuid: "af7bb239a5d6454daa140682a17b532d" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 43367 } } peers { permanent_uuid: "180459af951d4aabbb5312ba1ae741bb" member_type: NON_VOTER last_known_addr { host: "127.0.57.193" port: 37035 } attrs { } } } W0507 06:28:29.858572 1828 tablet_service.cc:646] failed transaction from {username='slave'} at 127.0.0.1:58072: Illegal state: Replica f9f586dd86b04fe984968b9de7f52209 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 1 leader_uuid: "af7bb239a5d6454daa140682a17b532d" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f9f586dd86b04fe984968b9de7f52209" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 36235 } } peers { permanent_uuid: "af7bb239a5d6454daa140682a17b532d" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 43367 } } peers { permanent_uuid: "180459af951d4aabbb5312ba1ae741bb" member_type: NON_VOTER last_known_addr { host: "127.0.57.193" port: 37035 } attrs { } } } W0507 06:28:29.873262 1703 tablet_service.cc:646] failed transaction from {username='slave'} at 127.0.0.1:52052: Illegal state: Replica af7bb239a5d6454daa140682a17b532d is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f9f586dd86b04fe984968b9de7f52209" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 36235 } } peers { permanent_uuid: "af7bb239a5d6454daa140682a17b532d" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 43367 } } peers { permanent_uuid: "180459af951d4aabbb5312ba1ae741bb" member_type: NON_VOTER last_known_addr { host: "127.0.57.193" port: 37035 } attrs { } } } W0507 06:28:29.874037 1706 tablet_service.cc:646] failed transaction from {username='slave'} at 127.0.0.1:52052: Illegal state: Replica af7bb239a5d6454daa140682a17b532d is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f9f586dd86b04fe984968b9de7f52209" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 36235 } } peers { permanent_uuid: "af7bb239a5d6454daa140682a17b532d" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 43367 } } peers { permanent_uuid: "180459af951d4aabbb5312ba1ae741bb" member_type: NON_VOTER last_known_addr { host: "127.0.57.193" port: 37035 } attrs { } } } W0507 06:28:29.876565 1703 tablet_service.cc:646] failed transaction from {username='slave'} at 127.0.0.1:52052: Illegal state: Replica af7bb239a5d6454daa140682a17b532d is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f9f586dd86b04fe984968b9de7f52209" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 36235 } } peers { permanent_uuid: "af7bb239a5d6454daa140682a17b532d" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 43367 } } peers { permanent_uuid: "180459af951d4aabbb5312ba1ae741bb" member_type: NON_VOTER last_known_addr { host: "127.0.57.193" port: 37035 } attrs { } } } W0507 06:28:29.876734 1706 tablet_service.cc:646] failed transaction from {username='slave'} at 127.0.0.1:52052: Illegal state: Replica af7bb239a5d6454daa140682a17b532d is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f9f586dd86b04fe984968b9de7f52209" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 36235 } } peers { permanent_uuid: "af7bb239a5d6454daa140682a17b532d" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 43367 } } peers { permanent_uuid: "180459af951d4aabbb5312ba1ae741bb" member_type: NON_VOTER last_known_addr { host: "127.0.57.193" port: 37035 } attrs { } } } W0507 06:28:29.895143 1828 tablet_service.cc:646] failed transaction from {username='slave'} at 127.0.0.1:58072: Illegal state: Replica f9f586dd86b04fe984968b9de7f52209 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 1 leader_uuid: "af7bb239a5d6454daa140682a17b532d" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f9f586dd86b04fe984968b9de7f52209" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 36235 } } peers { permanent_uuid: "af7bb239a5d6454daa140682a17b532d" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 43367 } } peers { permanent_uuid: "180459af951d4aabbb5312ba1ae741bb" member_type: NON_VOTER last_known_addr { host: "127.0.57.193" port: 37035 } attrs { } } } W0507 06:28:29.896185 1827 tablet_service.cc:646] failed transaction from {username='slave'} at 127.0.0.1:58072: Illegal state: Replica f9f586dd86b04fe984968b9de7f52209 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 1 leader_uuid: "af7bb239a5d6454daa140682a17b532d" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f9f586dd86b04fe984968b9de7f52209" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 36235 } } peers { permanent_uuid: "af7bb239a5d6454daa140682a17b532d" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 43367 } } peers { permanent_uuid: "180459af951d4aabbb5312ba1ae741bb" member_type: NON_VOTER last_known_addr { host: "127.0.57.193" port: 37035 } attrs { } } } W0507 06:28:29.896399 1828 tablet_service.cc:646] failed transaction from {username='slave'} at 127.0.0.1:58072: Illegal state: Replica f9f586dd86b04fe984968b9de7f52209 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 1 leader_uuid: "af7bb239a5d6454daa140682a17b532d" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f9f586dd86b04fe984968b9de7f52209" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 36235 } } peers { permanent_uuid: "af7bb239a5d6454daa140682a17b532d" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 43367 } } peers { permanent_uuid: "180459af951d4aabbb5312ba1ae741bb" member_type: NON_VOTER last_known_addr { host: "127.0.57.193" port: 37035 } attrs { } } } W0507 06:28:29.898241 1828 tablet_service.cc:646] failed transaction from {username='slave'} at 127.0.0.1:58072: Illegal state: Replica f9f586dd86b04fe984968b9de7f52209 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 1 leader_uuid: "af7bb239a5d6454daa140682a17b532d" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f9f586dd86b04fe984968b9de7f52209" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 36235 } } peers { permanent_uuid: "af7bb239a5d6454daa140682a17b532d" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 43367 } } peers { permanent_uuid: "180459af951d4aabbb5312ba1ae741bb" member_type: NON_VOTER last_known_addr { host: "127.0.57.193" port: 37035 } attrs { } } } W0507 06:28:29.920660 1706 tablet_service.cc:646] failed transaction from {username='slave'} at 127.0.0.1:52052: Illegal state: Replica af7bb239a5d6454daa140682a17b532d is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f9f586dd86b04fe984968b9de7f52209" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 36235 } } peers { permanent_uuid: "af7bb239a5d6454daa140682a17b532d" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 43367 } } peers { permanent_uuid: "180459af951d4aabbb5312ba1ae741bb" member_type: NON_VOTER last_known_addr { host: "127.0.57.193" port: 37035 } attrs { } } } W0507 06:28:29.920992 1703 tablet_service.cc:646] failed transaction from {username='slave'} at 127.0.0.1:52052: Illegal state: Replica af7bb239a5d6454daa140682a17b532d is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f9f586dd86b04fe984968b9de7f52209" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 36235 } } peers { permanent_uuid: "af7bb239a5d6454daa140682a17b532d" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 43367 } } peers { permanent_uuid: "180459af951d4aabbb5312ba1ae741bb" member_type: NON_VOTER last_known_addr { host: "127.0.57.193" port: 37035 } attrs { } } } W0507 06:28:29.921306 1705 tablet_service.cc:646] failed transaction from {username='slave'} at 127.0.0.1:52052: Illegal state: Replica af7bb239a5d6454daa140682a17b532d is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f9f586dd86b04fe984968b9de7f52209" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 36235 } } peers { permanent_uuid: "af7bb239a5d6454daa140682a17b532d" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 43367 } } peers { permanent_uuid: "180459af951d4aabbb5312ba1ae741bb" member_type: NON_VOTER last_known_addr { host: "127.0.57.193" port: 37035 } attrs { } } } W0507 06:28:29.923521 1705 tablet_service.cc:646] failed transaction from {username='slave'} at 127.0.0.1:52052: Illegal state: Replica af7bb239a5d6454daa140682a17b532d is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f9f586dd86b04fe984968b9de7f52209" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 36235 } } peers { permanent_uuid: "af7bb239a5d6454daa140682a17b532d" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 43367 } } peers { permanent_uuid: "180459af951d4aabbb5312ba1ae741bb" member_type: NON_VOTER last_known_addr { host: "127.0.57.193" port: 37035 } attrs { } } } W0507 06:28:29.942855 1828 tablet_service.cc:646] failed transaction from {username='slave'} at 127.0.0.1:58072: Illegal state: Replica f9f586dd86b04fe984968b9de7f52209 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 1 leader_uuid: "af7bb239a5d6454daa140682a17b532d" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f9f586dd86b04fe984968b9de7f52209" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 36235 } } peers { permanent_uuid: "af7bb239a5d6454daa140682a17b532d" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 43367 } } peers { permanent_uuid: "180459af951d4aabbb5312ba1ae741bb" member_type: NON_VOTER last_known_addr { host: "127.0.57.193" port: 37035 } attrs { } } } W0507 06:28:29.943495 1827 tablet_service.cc:646] failed transaction from {username='slave'} at 127.0.0.1:58072: Illegal state: Replica f9f586dd86b04fe984968b9de7f52209 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 1 leader_uuid: "af7bb239a5d6454daa140682a17b532d" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f9f586dd86b04fe984968b9de7f52209" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 36235 } } peers { permanent_uuid: "af7bb239a5d6454daa140682a17b532d" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 43367 } } peers { permanent_uuid: "180459af951d4aabbb5312ba1ae741bb" member_type: NON_VOTER last_known_addr { host: "127.0.57.193" port: 37035 } attrs { } } } W0507 06:28:29.944689 1827 tablet_service.cc:646] failed transaction from {username='slave'} at 127.0.0.1:58072: Illegal state: Replica f9f586dd86b04fe984968b9de7f52209 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 1 leader_uuid: "af7bb239a5d6454daa140682a17b532d" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f9f586dd86b04fe984968b9de7f52209" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 36235 } } peers { permanent_uuid: "af7bb239a5d6454daa140682a17b532d" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 43367 } } peers { permanent_uuid: "180459af951d4aabbb5312ba1ae741bb" member_type: NON_VOTER last_known_addr { host: "127.0.57.193" port: 37035 } attrs { } } } W0507 06:28:29.945979 1827 tablet_service.cc:646] failed transaction from {username='slave'} at 127.0.0.1:58072: Illegal state: Replica f9f586dd86b04fe984968b9de7f52209 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 1 leader_uuid: "af7bb239a5d6454daa140682a17b532d" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f9f586dd86b04fe984968b9de7f52209" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 36235 } } peers { permanent_uuid: "af7bb239a5d6454daa140682a17b532d" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 43367 } } peers { permanent_uuid: "180459af951d4aabbb5312ba1ae741bb" member_type: NON_VOTER last_known_addr { host: "127.0.57.193" port: 37035 } attrs { } } } W0507 06:28:29.971434 1705 tablet_service.cc:646] failed transaction from {username='slave'} at 127.0.0.1:52052: Illegal state: Replica af7bb239a5d6454daa140682a17b532d is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f9f586dd86b04fe984968b9de7f52209" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 36235 } } peers { permanent_uuid: "af7bb239a5d6454daa140682a17b532d" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 43367 } } peers { permanent_uuid: "180459af951d4aabbb5312ba1ae741bb" member_type: NON_VOTER last_known_addr { host: "127.0.57.193" port: 37035 } attrs { } } } W0507 06:28:29.971724 1706 tablet_service.cc:646] failed transaction from {username='slave'} at 127.0.0.1:52052: Illegal state: Replica af7bb239a5d6454daa140682a17b532d is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f9f586dd86b04fe984968b9de7f52209" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 36235 } } peers { permanent_uuid: "af7bb239a5d6454daa140682a17b532d" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 43367 } } peers { permanent_uuid: "180459af951d4aabbb5312ba1ae741bb" member_type: NON_VOTER last_known_addr { host: "127.0.57.193" port: 37035 } attrs { } } } W0507 06:28:29.971932 1703 tablet_service.cc:646] failed transaction from {username='slave'} at 127.0.0.1:52052: Illegal state: Replica af7bb239a5d6454daa140682a17b532d is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f9f586dd86b04fe984968b9de7f52209" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 36235 } } peers { permanent_uuid: "af7bb239a5d6454daa140682a17b532d" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 43367 } } peers { permanent_uuid: "180459af951d4aabbb5312ba1ae741bb" member_type: NON_VOTER last_known_addr { host: "127.0.57.193" port: 37035 } attrs { } } } W0507 06:28:29.972498 1705 tablet_service.cc:646] failed transaction from {username='slave'} at 127.0.0.1:52052: Illegal state: Replica af7bb239a5d6454daa140682a17b532d is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f9f586dd86b04fe984968b9de7f52209" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 36235 } } peers { permanent_uuid: "af7bb239a5d6454daa140682a17b532d" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 43367 } } peers { permanent_uuid: "180459af951d4aabbb5312ba1ae741bb" member_type: NON_VOTER last_known_addr { host: "127.0.57.193" port: 37035 } attrs { } } } W0507 06:28:29.997236 1827 tablet_service.cc:646] failed transaction from {username='slave'} at 127.0.0.1:58072: Illegal state: Replica f9f586dd86b04fe984968b9de7f52209 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 1 leader_uuid: "af7bb239a5d6454daa140682a17b532d" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f9f586dd86b04fe984968b9de7f52209" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 36235 } } peers { permanent_uuid: "af7bb239a5d6454daa140682a17b532d" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 43367 } } peers { permanent_uuid: "180459af951d4aabbb5312ba1ae741bb" member_type: NON_VOTER last_known_addr { host: "127.0.57.193" port: 37035 } attrs { } } } W0507 06:28:29.997964 1828 tablet_service.cc:646] failed transaction from {username='slave'} at 127.0.0.1:58072: Illegal state: Replica f9f586dd86b04fe984968b9de7f52209 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 1 leader_uuid: "af7bb239a5d6454daa140682a17b532d" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f9f586dd86b04fe984968b9de7f52209" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 36235 } } peers { permanent_uuid: "af7bb239a5d6454daa140682a17b532d" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 43367 } } peers { permanent_uuid: "180459af951d4aabbb5312ba1ae741bb" member_type: NON_VOTER last_known_addr { host: "127.0.57.193" port: 37035 } attrs { } } } W0507 06:28:29.999426 1827 tablet_service.cc:646] failed transaction from {username='slave'} at 127.0.0.1:58072: Illegal state: Replica f9f586dd86b04fe984968b9de7f52209 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 1 leader_uuid: "af7bb239a5d6454daa140682a17b532d" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f9f586dd86b04fe984968b9de7f52209" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 36235 } } peers { permanent_uuid: "af7bb239a5d6454daa140682a17b532d" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 43367 } } peers { permanent_uuid: "180459af951d4aabbb5312ba1ae741bb" member_type: NON_VOTER last_known_addr { host: "127.0.57.193" port: 37035 } attrs { } } } W0507 06:28:29.999632 1828 tablet_service.cc:646] failed transaction from {username='slave'} at 127.0.0.1:58072: Illegal state: Replica f9f586dd86b04fe984968b9de7f52209 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 1 leader_uuid: "af7bb239a5d6454daa140682a17b532d" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f9f586dd86b04fe984968b9de7f52209" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 36235 } } peers { permanent_uuid: "af7bb239a5d6454daa140682a17b532d" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 43367 } } peers { permanent_uuid: "180459af951d4aabbb5312ba1ae741bb" member_type: NON_VOTER last_known_addr { host: "127.0.57.193" port: 37035 } attrs { } } } W0507 06:28:30.025929 1705 tablet_service.cc:646] failed transaction from {username='slave'} at 127.0.0.1:52052: Illegal state: Replica af7bb239a5d6454daa140682a17b532d is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f9f586dd86b04fe984968b9de7f52209" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 36235 } } peers { permanent_uuid: "af7bb239a5d6454daa140682a17b532d" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 43367 } } peers { permanent_uuid: "180459af951d4aabbb5312ba1ae741bb" member_type: NON_VOTER last_known_addr { host: "127.0.57.193" port: 37035 } attrs { } } } W0507 06:28:30.027616 1705 tablet_service.cc:646] failed transaction from {username='slave'} at 127.0.0.1:52052: Illegal state: Replica af7bb239a5d6454daa140682a17b532d is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f9f586dd86b04fe984968b9de7f52209" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 36235 } } peers { permanent_uuid: "af7bb239a5d6454daa140682a17b532d" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 43367 } } peers { permanent_uuid: "180459af951d4aabbb5312ba1ae741bb" member_type: NON_VOTER last_known_addr { host: "127.0.57.193" port: 37035 } attrs { } } } W0507 06:28:30.027789 1706 tablet_service.cc:646] failed transaction from {username='slave'} at 127.0.0.1:52052: Illegal state: Replica af7bb239a5d6454daa140682a17b532d is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f9f586dd86b04fe984968b9de7f52209" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 36235 } } peers { permanent_uuid: "af7bb239a5d6454daa140682a17b532d" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 43367 } } peers { permanent_uuid: "180459af951d4aabbb5312ba1ae741bb" member_type: NON_VOTER last_known_addr { host: "127.0.57.193" port: 37035 } attrs { } } } W0507 06:28:30.027984 1703 tablet_service.cc:646] failed transaction from {username='slave'} at 127.0.0.1:52052: Illegal state: Replica af7bb239a5d6454daa140682a17b532d is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f9f586dd86b04fe984968b9de7f52209" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 36235 } } peers { permanent_uuid: "af7bb239a5d6454daa140682a17b532d" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 43367 } } peers { permanent_uuid: "180459af951d4aabbb5312ba1ae741bb" member_type: NON_VOTER last_known_addr { host: "127.0.57.193" port: 37035 } attrs { } } } W0507 06:28:30.053308 1828 tablet_service.cc:646] failed transaction from {username='slave'} at 127.0.0.1:58072: Illegal state: Replica f9f586dd86b04fe984968b9de7f52209 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 1 leader_uuid: "af7bb239a5d6454daa140682a17b532d" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f9f586dd86b04fe984968b9de7f52209" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 36235 } } peers { permanent_uuid: "af7bb239a5d6454daa140682a17b532d" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 43367 } } peers { permanent_uuid: "180459af951d4aabbb5312ba1ae741bb" member_type: NON_VOTER last_known_addr { host: "127.0.57.193" port: 37035 } attrs { } } } W0507 06:28:30.055388 1828 tablet_service.cc:646] failed transaction from {username='slave'} at 127.0.0.1:58072: Illegal state: Replica f9f586dd86b04fe984968b9de7f52209 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 1 leader_uuid: "af7bb239a5d6454daa140682a17b532d" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f9f586dd86b04fe984968b9de7f52209" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 36235 } } peers { permanent_uuid: "af7bb239a5d6454daa140682a17b532d" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 43367 } } peers { permanent_uuid: "180459af951d4aabbb5312ba1ae741bb" member_type: NON_VOTER last_known_addr { host: "127.0.57.193" port: 37035 } attrs { } } } W0507 06:28:30.055609 1827 tablet_service.cc:646] failed transaction from {username='slave'} at 127.0.0.1:58072: Illegal state: Replica f9f586dd86b04fe984968b9de7f52209 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 1 leader_uuid: "af7bb239a5d6454daa140682a17b532d" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f9f586dd86b04fe984968b9de7f52209" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 36235 } } peers { permanent_uuid: "af7bb239a5d6454daa140682a17b532d" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 43367 } } peers { permanent_uuid: "180459af951d4aabbb5312ba1ae741bb" member_type: NON_VOTER last_known_addr { host: "127.0.57.193" port: 37035 } attrs { } } } W0507 06:28:30.056550 1827 tablet_service.cc:646] failed transaction from {username='slave'} at 127.0.0.1:58072: Illegal state: Replica f9f586dd86b04fe984968b9de7f52209 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 1 leader_uuid: "af7bb239a5d6454daa140682a17b532d" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f9f586dd86b04fe984968b9de7f52209" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 36235 } } peers { permanent_uuid: "af7bb239a5d6454daa140682a17b532d" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 43367 } } peers { permanent_uuid: "180459af951d4aabbb5312ba1ae741bb" member_type: NON_VOTER last_known_addr { host: "127.0.57.193" port: 37035 } attrs { } } } W0507 06:28:30.083757 1703 tablet_service.cc:646] failed transaction from {username='slave'} at 127.0.0.1:52052: Illegal state: Replica af7bb239a5d6454daa140682a17b532d is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f9f586dd86b04fe984968b9de7f52209" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 36235 } } peers { permanent_uuid: "af7bb239a5d6454daa140682a17b532d" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 43367 } } peers { permanent_uuid: "180459af951d4aabbb5312ba1ae741bb" member_type: NON_VOTER last_known_addr { host: "127.0.57.193" port: 37035 } attrs { } } } W0507 06:28:30.084471 1706 tablet_service.cc:646] failed transaction from {username='slave'} at 127.0.0.1:52052: Illegal state: Replica af7bb239a5d6454daa140682a17b532d is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f9f586dd86b04fe984968b9de7f52209" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 36235 } } peers { permanent_uuid: "af7bb239a5d6454daa140682a17b532d" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 43367 } } peers { permanent_uuid: "180459af951d4aabbb5312ba1ae741bb" member_type: NON_VOTER last_known_addr { host: "127.0.57.193" port: 37035 } attrs { } } } W0507 06:28:30.085633 1706 tablet_service.cc:646] failed transaction from {username='slave'} at 127.0.0.1:52052: Illegal state: Replica af7bb239a5d6454daa140682a17b532d is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f9f586dd86b04fe984968b9de7f52209" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 36235 } } peers { permanent_uuid: "af7bb239a5d6454daa140682a17b532d" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 43367 } } peers { permanent_uuid: "180459af951d4aabbb5312ba1ae741bb" member_type: NON_VOTER last_known_addr { host: "127.0.57.193" port: 37035 } attrs { } } } W0507 06:28:30.086536 1706 tablet_service.cc:646] failed transaction from {username='slave'} at 127.0.0.1:52052: Illegal state: Replica af7bb239a5d6454daa140682a17b532d is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f9f586dd86b04fe984968b9de7f52209" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 36235 } } peers { permanent_uuid: "af7bb239a5d6454daa140682a17b532d" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 43367 } } peers { permanent_uuid: "180459af951d4aabbb5312ba1ae741bb" member_type: NON_VOTER last_known_addr { host: "127.0.57.193" port: 37035 } attrs { } } } W0507 06:28:30.114048 1827 tablet_service.cc:646] failed transaction from {username='slave'} at 127.0.0.1:58072: Illegal state: Replica f9f586dd86b04fe984968b9de7f52209 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 1 leader_uuid: "af7bb239a5d6454daa140682a17b532d" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f9f586dd86b04fe984968b9de7f52209" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 36235 } } peers { permanent_uuid: "af7bb239a5d6454daa140682a17b532d" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 43367 } } peers { permanent_uuid: "180459af951d4aabbb5312ba1ae741bb" member_type: NON_VOTER last_known_addr { host: "127.0.57.193" port: 37035 } attrs { } } } W0507 06:28:30.114722 1828 tablet_service.cc:646] failed transaction from {username='slave'} at 127.0.0.1:58072: Illegal state: Replica f9f586dd86b04fe984968b9de7f52209 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 1 leader_uuid: "af7bb239a5d6454daa140682a17b532d" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f9f586dd86b04fe984968b9de7f52209" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 36235 } } peers { permanent_uuid: "af7bb239a5d6454daa140682a17b532d" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 43367 } } peers { permanent_uuid: "180459af951d4aabbb5312ba1ae741bb" member_type: NON_VOTER last_known_addr { host: "127.0.57.193" port: 37035 } attrs { } } } W0507 06:28:30.115010 1826 tablet_service.cc:646] failed transaction from {username='slave'} at 127.0.0.1:58072: Illegal state: Replica f9f586dd86b04fe984968b9de7f52209 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 1 leader_uuid: "af7bb239a5d6454daa140682a17b532d" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f9f586dd86b04fe984968b9de7f52209" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 36235 } } peers { permanent_uuid: "af7bb239a5d6454daa140682a17b532d" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 43367 } } peers { permanent_uuid: "180459af951d4aabbb5312ba1ae741bb" member_type: NON_VOTER last_known_addr { host: "127.0.57.193" port: 37035 } attrs { } } } W0507 06:28:30.116129 1828 tablet_service.cc:646] failed transaction from {username='slave'} at 127.0.0.1:58072: Illegal state: Replica f9f586dd86b04fe984968b9de7f52209 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 1 leader_uuid: "af7bb239a5d6454daa140682a17b532d" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f9f586dd86b04fe984968b9de7f52209" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 36235 } } peers { permanent_uuid: "af7bb239a5d6454daa140682a17b532d" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 43367 } } peers { permanent_uuid: "180459af951d4aabbb5312ba1ae741bb" member_type: NON_VOTER last_known_addr { host: "127.0.57.193" port: 37035 } attrs { } } } W0507 06:28:30.144992 1706 tablet_service.cc:646] failed transaction from {username='slave'} at 127.0.0.1:52052: Illegal state: Replica af7bb239a5d6454daa140682a17b532d is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f9f586dd86b04fe984968b9de7f52209" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 36235 } } peers { permanent_uuid: "af7bb239a5d6454daa140682a17b532d" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 43367 } } peers { permanent_uuid: "180459af951d4aabbb5312ba1ae741bb" member_type: NON_VOTER last_known_addr { host: "127.0.57.193" port: 37035 } attrs { } } } W0507 06:28:30.145768 1703 tablet_service.cc:646] failed transaction from {username='slave'} at 127.0.0.1:52052: Illegal state: Replica af7bb239a5d6454daa140682a17b532d is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f9f586dd86b04fe984968b9de7f52209" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 36235 } } peers { permanent_uuid: "af7bb239a5d6454daa140682a17b532d" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 43367 } } peers { permanent_uuid: "180459af951d4aabbb5312ba1ae741bb" member_type: NON_VOTER last_known_addr { host: "127.0.57.193" port: 37035 } attrs { } } } W0507 06:28:30.148298 1703 tablet_service.cc:646] failed transaction from {username='slave'} at 127.0.0.1:52052: Illegal state: Replica af7bb239a5d6454daa140682a17b532d is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f9f586dd86b04fe984968b9de7f52209" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 36235 } } peers { permanent_uuid: "af7bb239a5d6454daa140682a17b532d" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 43367 } } peers { permanent_uuid: "180459af951d4aabbb5312ba1ae741bb" member_type: NON_VOTER last_known_addr { host: "127.0.57.193" port: 37035 } attrs { } } } W0507 06:28:30.148643 1706 tablet_service.cc:646] failed transaction from {username='slave'} at 127.0.0.1:52052: Illegal state: Replica af7bb239a5d6454daa140682a17b532d is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f9f586dd86b04fe984968b9de7f52209" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 36235 } } peers { permanent_uuid: "af7bb239a5d6454daa140682a17b532d" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 43367 } } peers { permanent_uuid: "180459af951d4aabbb5312ba1ae741bb" member_type: NON_VOTER last_known_addr { host: "127.0.57.193" port: 37035 } attrs { } } } W0507 06:28:30.177922 1828 tablet_service.cc:646] failed transaction from {username='slave'} at 127.0.0.1:58072: Illegal state: Replica f9f586dd86b04fe984968b9de7f52209 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 1 leader_uuid: "af7bb239a5d6454daa140682a17b532d" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f9f586dd86b04fe984968b9de7f52209" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 36235 } } peers { permanent_uuid: "af7bb239a5d6454daa140682a17b532d" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 43367 } } peers { permanent_uuid: "180459af951d4aabbb5312ba1ae741bb" member_type: NON_VOTER last_known_addr { host: "127.0.57.193" port: 37035 } attrs { } } } W0507 06:28:30.178807 1826 tablet_service.cc:646] failed transaction from {username='slave'} at 127.0.0.1:58072: Illegal state: Replica f9f586dd86b04fe984968b9de7f52209 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 1 leader_uuid: "af7bb239a5d6454daa140682a17b532d" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f9f586dd86b04fe984968b9de7f52209" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 36235 } } peers { permanent_uuid: "af7bb239a5d6454daa140682a17b532d" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 43367 } } peers { permanent_uuid: "180459af951d4aabbb5312ba1ae741bb" member_type: NON_VOTER last_known_addr { host: "127.0.57.193" port: 37035 } attrs { } } } W0507 06:28:30.179025 1828 tablet_service.cc:646] failed transaction from {username='slave'} at 127.0.0.1:58072: Illegal state: Replica f9f586dd86b04fe984968b9de7f52209 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 1 leader_uuid: "af7bb239a5d6454daa140682a17b532d" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f9f586dd86b04fe984968b9de7f52209" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 36235 } } peers { permanent_uuid: "af7bb239a5d6454daa140682a17b532d" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 43367 } } peers { permanent_uuid: "180459af951d4aabbb5312ba1ae741bb" member_type: NON_VOTER last_known_addr { host: "127.0.57.193" port: 37035 } attrs { } } } W0507 06:28:30.179868 1828 tablet_service.cc:646] failed transaction from {username='slave'} at 127.0.0.1:58072: Illegal state: Replica f9f586dd86b04fe984968b9de7f52209 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 1 leader_uuid: "af7bb239a5d6454daa140682a17b532d" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f9f586dd86b04fe984968b9de7f52209" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 36235 } } peers { permanent_uuid: "af7bb239a5d6454daa140682a17b532d" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 43367 } } peers { permanent_uuid: "180459af951d4aabbb5312ba1ae741bb" member_type: NON_VOTER last_known_addr { host: "127.0.57.193" port: 37035 } attrs { } } } W0507 06:28:30.210955 1706 tablet_service.cc:646] failed transaction from {username='slave'} at 127.0.0.1:52052: Illegal state: Replica af7bb239a5d6454daa140682a17b532d is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f9f586dd86b04fe984968b9de7f52209" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 36235 } } peers { permanent_uuid: "af7bb239a5d6454daa140682a17b532d" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 43367 } } peers { permanent_uuid: "180459af951d4aabbb5312ba1ae741bb" member_type: NON_VOTER last_known_addr { host: "127.0.57.193" port: 37035 } attrs { } } } W0507 06:28:30.211735 1703 tablet_service.cc:646] failed transaction from {username='slave'} at 127.0.0.1:52052: Illegal state: Replica af7bb239a5d6454daa140682a17b532d is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f9f586dd86b04fe984968b9de7f52209" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 36235 } } peers { permanent_uuid: "af7bb239a5d6454daa140682a17b532d" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 43367 } } peers { permanent_uuid: "180459af951d4aabbb5312ba1ae741bb" member_type: NON_VOTER last_known_addr { host: "127.0.57.193" port: 37035 } attrs { } } } W0507 06:28:30.213915 1703 tablet_service.cc:646] failed transaction from {username='slave'} at 127.0.0.1:52052: Illegal state: Replica af7bb239a5d6454daa140682a17b532d is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f9f586dd86b04fe984968b9de7f52209" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 36235 } } peers { permanent_uuid: "af7bb239a5d6454daa140682a17b532d" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 43367 } } peers { permanent_uuid: "180459af951d4aabbb5312ba1ae741bb" member_type: NON_VOTER last_known_addr { host: "127.0.57.193" port: 37035 } attrs { } } } W0507 06:28:30.214040 1706 tablet_service.cc:646] failed transaction from {username='slave'} at 127.0.0.1:52052: Illegal state: Replica af7bb239a5d6454daa140682a17b532d is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f9f586dd86b04fe984968b9de7f52209" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 36235 } } peers { permanent_uuid: "af7bb239a5d6454daa140682a17b532d" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 43367 } } peers { permanent_uuid: "180459af951d4aabbb5312ba1ae741bb" member_type: NON_VOTER last_known_addr { host: "127.0.57.193" port: 37035 } attrs { } } } W0507 06:28:30.243393 1826 tablet_service.cc:646] failed transaction from {username='slave'} at 127.0.0.1:58072: Illegal state: Replica f9f586dd86b04fe984968b9de7f52209 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 1 leader_uuid: "af7bb239a5d6454daa140682a17b532d" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f9f586dd86b04fe984968b9de7f52209" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 36235 } } peers { permanent_uuid: "af7bb239a5d6454daa140682a17b532d" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 43367 } } peers { permanent_uuid: "180459af951d4aabbb5312ba1ae741bb" member_type: NON_VOTER last_known_addr { host: "127.0.57.193" port: 37035 } attrs { } } } W0507 06:28:30.243659 1828 tablet_service.cc:646] failed transaction from {username='slave'} at 127.0.0.1:58072: Illegal state: Replica f9f586dd86b04fe984968b9de7f52209 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 1 leader_uuid: "af7bb239a5d6454daa140682a17b532d" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f9f586dd86b04fe984968b9de7f52209" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 36235 } } peers { permanent_uuid: "af7bb239a5d6454daa140682a17b532d" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 43367 } } peers { permanent_uuid: "180459af951d4aabbb5312ba1ae741bb" member_type: NON_VOTER last_known_addr { host: "127.0.57.193" port: 37035 } attrs { } } } W0507 06:28:30.246135 1828 tablet_service.cc:646] failed transaction from {username='slave'} at 127.0.0.1:58072: Illegal state: Replica f9f586dd86b04fe984968b9de7f52209 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 1 leader_uuid: "af7bb239a5d6454daa140682a17b532d" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f9f586dd86b04fe984968b9de7f52209" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 36235 } } peers { permanent_uuid: "af7bb239a5d6454daa140682a17b532d" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 43367 } } peers { permanent_uuid: "180459af951d4aabbb5312ba1ae741bb" member_type: NON_VOTER last_known_addr { host: "127.0.57.193" port: 37035 } attrs { } } } W0507 06:28:30.247319 1828 tablet_service.cc:646] failed transaction from {username='slave'} at 127.0.0.1:58072: Illegal state: Replica f9f586dd86b04fe984968b9de7f52209 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 1 leader_uuid: "af7bb239a5d6454daa140682a17b532d" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f9f586dd86b04fe984968b9de7f52209" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 36235 } } peers { permanent_uuid: "af7bb239a5d6454daa140682a17b532d" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 43367 } } peers { permanent_uuid: "180459af951d4aabbb5312ba1ae741bb" member_type: NON_VOTER last_known_addr { host: "127.0.57.193" port: 37035 } attrs { } } } W0507 06:28:30.279420 1706 tablet_service.cc:646] failed transaction from {username='slave'} at 127.0.0.1:52052: Illegal state: Replica af7bb239a5d6454daa140682a17b532d is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f9f586dd86b04fe984968b9de7f52209" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 36235 } } peers { permanent_uuid: "af7bb239a5d6454daa140682a17b532d" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 43367 } } peers { permanent_uuid: "180459af951d4aabbb5312ba1ae741bb" member_type: NON_VOTER last_known_addr { host: "127.0.57.193" port: 37035 } attrs { } } } W0507 06:28:30.279695 1703 tablet_service.cc:646] failed transaction from {username='slave'} at 127.0.0.1:52052: Illegal state: Replica af7bb239a5d6454daa140682a17b532d is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f9f586dd86b04fe984968b9de7f52209" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 36235 } } peers { permanent_uuid: "af7bb239a5d6454daa140682a17b532d" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 43367 } } peers { permanent_uuid: "180459af951d4aabbb5312ba1ae741bb" member_type: NON_VOTER last_known_addr { host: "127.0.57.193" port: 37035 } attrs { } } } W0507 06:28:30.280037 1705 tablet_service.cc:646] failed transaction from {username='slave'} at 127.0.0.1:52052: Illegal state: Replica af7bb239a5d6454daa140682a17b532d is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f9f586dd86b04fe984968b9de7f52209" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 36235 } } peers { permanent_uuid: "af7bb239a5d6454daa140682a17b532d" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 43367 } } peers { permanent_uuid: "180459af951d4aabbb5312ba1ae741bb" member_type: NON_VOTER last_known_addr { host: "127.0.57.193" port: 37035 } attrs { } } } W0507 06:28:30.283848 1705 tablet_service.cc:646] failed transaction from {username='slave'} at 127.0.0.1:52052: Illegal state: Replica af7bb239a5d6454daa140682a17b532d is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f9f586dd86b04fe984968b9de7f52209" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 36235 } } peers { permanent_uuid: "af7bb239a5d6454daa140682a17b532d" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 43367 } } peers { permanent_uuid: "180459af951d4aabbb5312ba1ae741bb" member_type: NON_VOTER last_known_addr { host: "127.0.57.193" port: 37035 } attrs { } } } W0507 06:28:30.316258 1828 tablet_service.cc:646] failed transaction from {username='slave'} at 127.0.0.1:58072: Illegal state: Replica f9f586dd86b04fe984968b9de7f52209 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 1 leader_uuid: "af7bb239a5d6454daa140682a17b532d" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f9f586dd86b04fe984968b9de7f52209" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 36235 } } peers { permanent_uuid: "af7bb239a5d6454daa140682a17b532d" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 43367 } } peers { permanent_uuid: "180459af951d4aabbb5312ba1ae741bb" member_type: NON_VOTER last_known_addr { host: "127.0.57.193" port: 37035 } attrs { } } } W0507 06:28:30.316578 1826 tablet_service.cc:646] failed transaction from {username='slave'} at 127.0.0.1:58072: Illegal state: Replica f9f586dd86b04fe984968b9de7f52209 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 1 leader_uuid: "af7bb239a5d6454daa140682a17b532d" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f9f586dd86b04fe984968b9de7f52209" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 36235 } } peers { permanent_uuid: "af7bb239a5d6454daa140682a17b532d" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 43367 } } peers { permanent_uuid: "180459af951d4aabbb5312ba1ae741bb" member_type: NON_VOTER last_known_addr { host: "127.0.57.193" port: 37035 } attrs { } } } W0507 06:28:30.317247 1827 tablet_service.cc:646] failed transaction from {username='slave'} at 127.0.0.1:58072: Illegal state: Replica f9f586dd86b04fe984968b9de7f52209 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 1 leader_uuid: "af7bb239a5d6454daa140682a17b532d" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f9f586dd86b04fe984968b9de7f52209" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 36235 } } peers { permanent_uuid: "af7bb239a5d6454daa140682a17b532d" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 43367 } } peers { permanent_uuid: "180459af951d4aabbb5312ba1ae741bb" member_type: NON_VOTER last_known_addr { host: "127.0.57.193" port: 37035 } attrs { } } } W0507 06:28:30.317646 1826 tablet_service.cc:646] failed transaction from {username='slave'} at 127.0.0.1:58072: Illegal state: Replica f9f586dd86b04fe984968b9de7f52209 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 1 leader_uuid: "af7bb239a5d6454daa140682a17b532d" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f9f586dd86b04fe984968b9de7f52209" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 36235 } } peers { permanent_uuid: "af7bb239a5d6454daa140682a17b532d" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 43367 } } peers { permanent_uuid: "180459af951d4aabbb5312ba1ae741bb" member_type: NON_VOTER last_known_addr { host: "127.0.57.193" port: 37035 } attrs { } } } W0507 06:28:30.355690 1705 tablet_service.cc:646] failed transaction from {username='slave'} at 127.0.0.1:52052: Illegal state: Replica af7bb239a5d6454daa140682a17b532d is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f9f586dd86b04fe984968b9de7f52209" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 36235 } } peers { permanent_uuid: "af7bb239a5d6454daa140682a17b532d" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 43367 } } peers { permanent_uuid: "180459af951d4aabbb5312ba1ae741bb" member_type: NON_VOTER last_known_addr { host: "127.0.57.193" port: 37035 } attrs { } } } W0507 06:28:30.356148 1703 tablet_service.cc:646] failed transaction from {username='slave'} at 127.0.0.1:52052: Illegal state: Replica af7bb239a5d6454daa140682a17b532d is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f9f586dd86b04fe984968b9de7f52209" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 36235 } } peers { permanent_uuid: "af7bb239a5d6454daa140682a17b532d" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 43367 } } peers { permanent_uuid: "180459af951d4aabbb5312ba1ae741bb" member_type: NON_VOTER last_known_addr { host: "127.0.57.193" port: 37035 } attrs { } } } W0507 06:28:30.357061 1703 tablet_service.cc:646] failed transaction from {username='slave'} at 127.0.0.1:52052: Illegal state: Replica af7bb239a5d6454daa140682a17b532d is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f9f586dd86b04fe984968b9de7f52209" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 36235 } } peers { permanent_uuid: "af7bb239a5d6454daa140682a17b532d" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 43367 } } peers { permanent_uuid: "180459af951d4aabbb5312ba1ae741bb" member_type: NON_VOTER last_known_addr { host: "127.0.57.193" port: 37035 } attrs { } } } W0507 06:28:30.357317 1705 tablet_service.cc:646] failed transaction from {username='slave'} at 127.0.0.1:52052: Illegal state: Replica af7bb239a5d6454daa140682a17b532d is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f9f586dd86b04fe984968b9de7f52209" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 36235 } } peers { permanent_uuid: "af7bb239a5d6454daa140682a17b532d" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 43367 } } peers { permanent_uuid: "180459af951d4aabbb5312ba1ae741bb" member_type: NON_VOTER last_known_addr { host: "127.0.57.193" port: 37035 } attrs { } } } W0507 06:28:30.393007 1826 tablet_service.cc:646] failed transaction from {username='slave'} at 127.0.0.1:58072: Illegal state: Replica f9f586dd86b04fe984968b9de7f52209 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 1 leader_uuid: "af7bb239a5d6454daa140682a17b532d" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f9f586dd86b04fe984968b9de7f52209" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 36235 } } peers { permanent_uuid: "af7bb239a5d6454daa140682a17b532d" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 43367 } } peers { permanent_uuid: "180459af951d4aabbb5312ba1ae741bb" member_type: NON_VOTER last_known_addr { host: "127.0.57.193" port: 37035 } attrs { } } } W0507 06:28:30.393780 1827 tablet_service.cc:646] failed transaction from {username='slave'} at 127.0.0.1:58072: Illegal state: Replica f9f586dd86b04fe984968b9de7f52209 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 1 leader_uuid: "af7bb239a5d6454daa140682a17b532d" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f9f586dd86b04fe984968b9de7f52209" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 36235 } } peers { permanent_uuid: "af7bb239a5d6454daa140682a17b532d" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 43367 } } peers { permanent_uuid: "180459af951d4aabbb5312ba1ae741bb" member_type: NON_VOTER last_known_addr { host: "127.0.57.193" port: 37035 } attrs { } } } W0507 06:28:30.394057 1828 tablet_service.cc:646] failed transaction from {username='slave'} at 127.0.0.1:58072: Illegal state: Replica f9f586dd86b04fe984968b9de7f52209 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 1 leader_uuid: "af7bb239a5d6454daa140682a17b532d" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f9f586dd86b04fe984968b9de7f52209" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 36235 } } peers { permanent_uuid: "af7bb239a5d6454daa140682a17b532d" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 43367 } } peers { permanent_uuid: "180459af951d4aabbb5312ba1ae741bb" member_type: NON_VOTER last_known_addr { host: "127.0.57.193" port: 37035 } attrs { } } } W0507 06:28:30.395100 1828 tablet_service.cc:646] failed transaction from {username='slave'} at 127.0.0.1:58072: Illegal state: Replica f9f586dd86b04fe984968b9de7f52209 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 1 leader_uuid: "af7bb239a5d6454daa140682a17b532d" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f9f586dd86b04fe984968b9de7f52209" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 36235 } } peers { permanent_uuid: "af7bb239a5d6454daa140682a17b532d" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 43367 } } peers { permanent_uuid: "180459af951d4aabbb5312ba1ae741bb" member_type: NON_VOTER last_known_addr { host: "127.0.57.193" port: 37035 } attrs { } } } W0507 06:28:30.435663 1705 tablet_service.cc:646] failed transaction from {username='slave'} at 127.0.0.1:52052: Illegal state: Replica af7bb239a5d6454daa140682a17b532d is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f9f586dd86b04fe984968b9de7f52209" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 36235 } } peers { permanent_uuid: "af7bb239a5d6454daa140682a17b532d" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 43367 } } peers { permanent_uuid: "180459af951d4aabbb5312ba1ae741bb" member_type: NON_VOTER last_known_addr { host: "127.0.57.193" port: 37035 } attrs { } } } W0507 06:28:30.435842 1703 tablet_service.cc:646] failed transaction from {username='slave'} at 127.0.0.1:52052: Illegal state: Replica af7bb239a5d6454daa140682a17b532d is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f9f586dd86b04fe984968b9de7f52209" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 36235 } } peers { permanent_uuid: "af7bb239a5d6454daa140682a17b532d" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 43367 } } peers { permanent_uuid: "180459af951d4aabbb5312ba1ae741bb" member_type: NON_VOTER last_known_addr { host: "127.0.57.193" port: 37035 } attrs { } } } W0507 06:28:30.436946 1705 tablet_service.cc:646] failed transaction from {username='slave'} at 127.0.0.1:52052: Illegal state: Replica af7bb239a5d6454daa140682a17b532d is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f9f586dd86b04fe984968b9de7f52209" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 36235 } } peers { permanent_uuid: "af7bb239a5d6454daa140682a17b532d" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 43367 } } peers { permanent_uuid: "180459af951d4aabbb5312ba1ae741bb" member_type: NON_VOTER last_known_addr { host: "127.0.57.193" port: 37035 } attrs { } } } W0507 06:28:30.437130 1703 tablet_service.cc:646] failed transaction from {username='slave'} at 127.0.0.1:52052: Illegal state: Replica af7bb239a5d6454daa140682a17b532d is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f9f586dd86b04fe984968b9de7f52209" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 36235 } } peers { permanent_uuid: "af7bb239a5d6454daa140682a17b532d" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 43367 } } peers { permanent_uuid: "180459af951d4aabbb5312ba1ae741bb" member_type: NON_VOTER last_known_addr { host: "127.0.57.193" port: 37035 } attrs { } } } W0507 06:28:30.474716 1827 tablet_service.cc:646] failed transaction from {username='slave'} at 127.0.0.1:58072: Illegal state: Replica f9f586dd86b04fe984968b9de7f52209 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 1 leader_uuid: "af7bb239a5d6454daa140682a17b532d" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f9f586dd86b04fe984968b9de7f52209" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 36235 } } peers { permanent_uuid: "af7bb239a5d6454daa140682a17b532d" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 43367 } } peers { permanent_uuid: "180459af951d4aabbb5312ba1ae741bb" member_type: NON_VOTER last_known_addr { host: "127.0.57.193" port: 37035 } attrs { } } } W0507 06:28:30.474957 1828 tablet_service.cc:646] failed transaction from {username='slave'} at 127.0.0.1:58072: Illegal state: Replica f9f586dd86b04fe984968b9de7f52209 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 1 leader_uuid: "af7bb239a5d6454daa140682a17b532d" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f9f586dd86b04fe984968b9de7f52209" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 36235 } } peers { permanent_uuid: "af7bb239a5d6454daa140682a17b532d" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 43367 } } peers { permanent_uuid: "180459af951d4aabbb5312ba1ae741bb" member_type: NON_VOTER last_known_addr { host: "127.0.57.193" port: 37035 } attrs { } } } W0507 06:28:30.475992 1826 tablet_service.cc:646] failed transaction from {username='slave'} at 127.0.0.1:58072: Illegal state: Replica f9f586dd86b04fe984968b9de7f52209 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 1 leader_uuid: "af7bb239a5d6454daa140682a17b532d" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f9f586dd86b04fe984968b9de7f52209" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 36235 } } peers { permanent_uuid: "af7bb239a5d6454daa140682a17b532d" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 43367 } } peers { permanent_uuid: "180459af951d4aabbb5312ba1ae741bb" member_type: NON_VOTER last_known_addr { host: "127.0.57.193" port: 37035 } attrs { } } } W0507 06:28:30.476191 1825 tablet_service.cc:646] failed transaction from {username='slave'} at 127.0.0.1:58072: Illegal state: Replica f9f586dd86b04fe984968b9de7f52209 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 1 leader_uuid: "af7bb239a5d6454daa140682a17b532d" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f9f586dd86b04fe984968b9de7f52209" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 36235 } } peers { permanent_uuid: "af7bb239a5d6454daa140682a17b532d" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 43367 } } peers { permanent_uuid: "180459af951d4aabbb5312ba1ae741bb" member_type: NON_VOTER last_known_addr { host: "127.0.57.193" port: 37035 } attrs { } } } W0507 06:28:30.516417 1703 tablet_service.cc:646] failed transaction from {username='slave'} at 127.0.0.1:52052: Illegal state: Replica af7bb239a5d6454daa140682a17b532d is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f9f586dd86b04fe984968b9de7f52209" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 36235 } } peers { permanent_uuid: "af7bb239a5d6454daa140682a17b532d" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 43367 } } peers { permanent_uuid: "180459af951d4aabbb5312ba1ae741bb" member_type: NON_VOTER last_known_addr { host: "127.0.57.193" port: 37035 } attrs { } } } W0507 06:28:30.517683 1703 tablet_service.cc:646] failed transaction from {username='slave'} at 127.0.0.1:52052: Illegal state: Replica af7bb239a5d6454daa140682a17b532d is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f9f586dd86b04fe984968b9de7f52209" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 36235 } } peers { permanent_uuid: "af7bb239a5d6454daa140682a17b532d" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 43367 } } peers { permanent_uuid: "180459af951d4aabbb5312ba1ae741bb" member_type: NON_VOTER last_known_addr { host: "127.0.57.193" port: 37035 } attrs { } } } W0507 06:28:30.517885 1705 tablet_service.cc:646] failed transaction from {username='slave'} at 127.0.0.1:52052: Illegal state: Replica af7bb239a5d6454daa140682a17b532d is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f9f586dd86b04fe984968b9de7f52209" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 36235 } } peers { permanent_uuid: "af7bb239a5d6454daa140682a17b532d" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 43367 } } peers { permanent_uuid: "180459af951d4aabbb5312ba1ae741bb" member_type: NON_VOTER last_known_addr { host: "127.0.57.193" port: 37035 } attrs { } } } W0507 06:28:30.518529 1706 tablet_service.cc:646] failed transaction from {username='slave'} at 127.0.0.1:52052: Illegal state: Replica af7bb239a5d6454daa140682a17b532d is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f9f586dd86b04fe984968b9de7f52209" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 36235 } } peers { permanent_uuid: "af7bb239a5d6454daa140682a17b532d" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 43367 } } peers { permanent_uuid: "180459af951d4aabbb5312ba1ae741bb" member_type: NON_VOTER last_known_addr { host: "127.0.57.193" port: 37035 } attrs { } } } W0507 06:28:30.559378 1826 tablet_service.cc:646] failed transaction from {username='slave'} at 127.0.0.1:58072: Illegal state: Replica f9f586dd86b04fe984968b9de7f52209 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 1 leader_uuid: "af7bb239a5d6454daa140682a17b532d" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f9f586dd86b04fe984968b9de7f52209" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 36235 } } peers { permanent_uuid: "af7bb239a5d6454daa140682a17b532d" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 43367 } } peers { permanent_uuid: "180459af951d4aabbb5312ba1ae741bb" member_type: NON_VOTER last_known_addr { host: "127.0.57.193" port: 37035 } attrs { } } } W0507 06:28:30.559621 1825 tablet_service.cc:646] failed transaction from {username='slave'} at 127.0.0.1:58072: Illegal state: Replica f9f586dd86b04fe984968b9de7f52209 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 1 leader_uuid: "af7bb239a5d6454daa140682a17b532d" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f9f586dd86b04fe984968b9de7f52209" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 36235 } } peers { permanent_uuid: "af7bb239a5d6454daa140682a17b532d" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 43367 } } peers { permanent_uuid: "180459af951d4aabbb5312ba1ae741bb" member_type: NON_VOTER last_known_addr { host: "127.0.57.193" port: 37035 } attrs { } } } W0507 06:28:30.560062 1827 tablet_service.cc:646] failed transaction from {username='slave'} at 127.0.0.1:58072: Illegal state: Replica f9f586dd86b04fe984968b9de7f52209 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 1 leader_uuid: "af7bb239a5d6454daa140682a17b532d" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f9f586dd86b04fe984968b9de7f52209" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 36235 } } peers { permanent_uuid: "af7bb239a5d6454daa140682a17b532d" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 43367 } } peers { permanent_uuid: "180459af951d4aabbb5312ba1ae741bb" member_type: NON_VOTER last_known_addr { host: "127.0.57.193" port: 37035 } attrs { } } } W0507 06:28:30.560356 1828 tablet_service.cc:646] failed transaction from {username='slave'} at 127.0.0.1:58072: Illegal state: Replica f9f586dd86b04fe984968b9de7f52209 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 1 leader_uuid: "af7bb239a5d6454daa140682a17b532d" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f9f586dd86b04fe984968b9de7f52209" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 36235 } } peers { permanent_uuid: "af7bb239a5d6454daa140682a17b532d" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 43367 } } peers { permanent_uuid: "180459af951d4aabbb5312ba1ae741bb" member_type: NON_VOTER last_known_addr { host: "127.0.57.193" port: 37035 } attrs { } } } W0507 06:28:30.605504 1706 tablet_service.cc:646] failed transaction from {username='slave'} at 127.0.0.1:52052: Illegal state: Replica af7bb239a5d6454daa140682a17b532d is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f9f586dd86b04fe984968b9de7f52209" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 36235 } } peers { permanent_uuid: "af7bb239a5d6454daa140682a17b532d" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 43367 } } peers { permanent_uuid: "180459af951d4aabbb5312ba1ae741bb" member_type: NON_VOTER last_known_addr { host: "127.0.57.193" port: 37035 } attrs { } } } W0507 06:28:30.606405 1703 tablet_service.cc:646] failed transaction from {username='slave'} at 127.0.0.1:52052: Illegal state: Replica af7bb239a5d6454daa140682a17b532d is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f9f586dd86b04fe984968b9de7f52209" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 36235 } } peers { permanent_uuid: "af7bb239a5d6454daa140682a17b532d" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 43367 } } peers { permanent_uuid: "180459af951d4aabbb5312ba1ae741bb" member_type: NON_VOTER last_known_addr { host: "127.0.57.193" port: 37035 } attrs { } } } W0507 06:28:30.606688 1705 tablet_service.cc:646] failed transaction from {username='slave'} at 127.0.0.1:52052: Illegal state: Replica af7bb239a5d6454daa140682a17b532d is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f9f586dd86b04fe984968b9de7f52209" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 36235 } } peers { permanent_uuid: "af7bb239a5d6454daa140682a17b532d" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 43367 } } peers { permanent_uuid: "180459af951d4aabbb5312ba1ae741bb" member_type: NON_VOTER last_known_addr { host: "127.0.57.193" port: 37035 } attrs { } } } W0507 06:28:30.607005 1706 tablet_service.cc:646] failed transaction from {username='slave'} at 127.0.0.1:52052: Illegal state: Replica af7bb239a5d6454daa140682a17b532d is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f9f586dd86b04fe984968b9de7f52209" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 36235 } } peers { permanent_uuid: "af7bb239a5d6454daa140682a17b532d" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 43367 } } peers { permanent_uuid: "180459af951d4aabbb5312ba1ae741bb" member_type: NON_VOTER last_known_addr { host: "127.0.57.193" port: 37035 } attrs { } } } W0507 06:28:30.648418 1827 tablet_service.cc:646] failed transaction from {username='slave'} at 127.0.0.1:58072: Illegal state: Replica f9f586dd86b04fe984968b9de7f52209 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 1 leader_uuid: "af7bb239a5d6454daa140682a17b532d" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f9f586dd86b04fe984968b9de7f52209" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 36235 } } peers { permanent_uuid: "af7bb239a5d6454daa140682a17b532d" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 43367 } } peers { permanent_uuid: "180459af951d4aabbb5312ba1ae741bb" member_type: NON_VOTER last_known_addr { host: "127.0.57.193" port: 37035 } attrs { } } } W0507 06:28:30.648633 1828 tablet_service.cc:646] failed transaction from {username='slave'} at 127.0.0.1:58072: Illegal state: Replica f9f586dd86b04fe984968b9de7f52209 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 1 leader_uuid: "af7bb239a5d6454daa140682a17b532d" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f9f586dd86b04fe984968b9de7f52209" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 36235 } } peers { permanent_uuid: "af7bb239a5d6454daa140682a17b532d" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 43367 } } peers { permanent_uuid: "180459af951d4aabbb5312ba1ae741bb" member_type: NON_VOTER last_known_addr { host: "127.0.57.193" port: 37035 } attrs { } } } W0507 06:28:30.649262 1825 tablet_service.cc:646] failed transaction from {username='slave'} at 127.0.0.1:58072: Illegal state: Replica f9f586dd86b04fe984968b9de7f52209 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 1 leader_uuid: "af7bb239a5d6454daa140682a17b532d" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f9f586dd86b04fe984968b9de7f52209" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 36235 } } peers { permanent_uuid: "af7bb239a5d6454daa140682a17b532d" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 43367 } } peers { permanent_uuid: "180459af951d4aabbb5312ba1ae741bb" member_type: NON_VOTER last_known_addr { host: "127.0.57.193" port: 37035 } attrs { } } } W0507 06:28:30.650429 1825 tablet_service.cc:646] failed transaction from {username='slave'} at 127.0.0.1:58072: Illegal state: Replica f9f586dd86b04fe984968b9de7f52209 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 1 leader_uuid: "af7bb239a5d6454daa140682a17b532d" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f9f586dd86b04fe984968b9de7f52209" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 36235 } } peers { permanent_uuid: "af7bb239a5d6454daa140682a17b532d" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 43367 } } peers { permanent_uuid: "180459af951d4aabbb5312ba1ae741bb" member_type: NON_VOTER last_known_addr { host: "127.0.57.193" port: 37035 } attrs { } } } W0507 06:28:30.693728 1706 tablet_service.cc:646] failed transaction from {username='slave'} at 127.0.0.1:52052: Illegal state: Replica af7bb239a5d6454daa140682a17b532d is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f9f586dd86b04fe984968b9de7f52209" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 36235 } } peers { permanent_uuid: "af7bb239a5d6454daa140682a17b532d" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 43367 } } peers { permanent_uuid: "180459af951d4aabbb5312ba1ae741bb" member_type: NON_VOTER last_known_addr { host: "127.0.57.193" port: 37035 } attrs { } } } W0507 06:28:30.695785 1706 tablet_service.cc:646] failed transaction from {username='slave'} at 127.0.0.1:52052: Illegal state: Replica af7bb239a5d6454daa140682a17b532d is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f9f586dd86b04fe984968b9de7f52209" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 36235 } } peers { permanent_uuid: "af7bb239a5d6454daa140682a17b532d" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 43367 } } peers { permanent_uuid: "180459af951d4aabbb5312ba1ae741bb" member_type: NON_VOTER last_known_addr { host: "127.0.57.193" port: 37035 } attrs { } } } W0507 06:28:30.695988 1705 tablet_service.cc:646] failed transaction from {username='slave'} at 127.0.0.1:52052: Illegal state: Replica af7bb239a5d6454daa140682a17b532d is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f9f586dd86b04fe984968b9de7f52209" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 36235 } } peers { permanent_uuid: "af7bb239a5d6454daa140682a17b532d" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 43367 } } peers { permanent_uuid: "180459af951d4aabbb5312ba1ae741bb" member_type: NON_VOTER last_known_addr { host: "127.0.57.193" port: 37035 } attrs { } } } W0507 06:28:30.696825 1705 tablet_service.cc:646] failed transaction from {username='slave'} at 127.0.0.1:52052: Illegal state: Replica af7bb239a5d6454daa140682a17b532d is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f9f586dd86b04fe984968b9de7f52209" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 36235 } } peers { permanent_uuid: "af7bb239a5d6454daa140682a17b532d" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 43367 } } peers { permanent_uuid: "180459af951d4aabbb5312ba1ae741bb" member_type: NON_VOTER last_known_addr { host: "127.0.57.193" port: 37035 } attrs { } } } W0507 06:28:30.738412 1825 tablet_service.cc:646] failed transaction from {username='slave'} at 127.0.0.1:58072: Illegal state: Replica f9f586dd86b04fe984968b9de7f52209 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 1 leader_uuid: "af7bb239a5d6454daa140682a17b532d" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f9f586dd86b04fe984968b9de7f52209" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 36235 } } peers { permanent_uuid: "af7bb239a5d6454daa140682a17b532d" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 43367 } } peers { permanent_uuid: "180459af951d4aabbb5312ba1ae741bb" member_type: NON_VOTER last_known_addr { host: "127.0.57.193" port: 37035 } attrs { } } } W0507 06:28:30.742417 1825 tablet_service.cc:646] failed transaction from {username='slave'} at 127.0.0.1:58072: Illegal state: Replica f9f586dd86b04fe984968b9de7f52209 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 1 leader_uuid: "af7bb239a5d6454daa140682a17b532d" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f9f586dd86b04fe984968b9de7f52209" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 36235 } } peers { permanent_uuid: "af7bb239a5d6454daa140682a17b532d" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 43367 } } peers { permanent_uuid: "180459af951d4aabbb5312ba1ae741bb" member_type: NON_VOTER last_known_addr { host: "127.0.57.193" port: 37035 } attrs { } } } W0507 06:28:30.743834 1825 tablet_service.cc:646] failed transaction from {username='slave'} at 127.0.0.1:58072: Illegal state: Replica f9f586dd86b04fe984968b9de7f52209 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 1 leader_uuid: "af7bb239a5d6454daa140682a17b532d" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f9f586dd86b04fe984968b9de7f52209" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 36235 } } peers { permanent_uuid: "af7bb239a5d6454daa140682a17b532d" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 43367 } } peers { permanent_uuid: "180459af951d4aabbb5312ba1ae741bb" member_type: NON_VOTER last_known_addr { host: "127.0.57.193" port: 37035 } attrs { } } } W0507 06:28:30.744009 1827 tablet_service.cc:646] failed transaction from {username='slave'} at 127.0.0.1:58072: Illegal state: Replica f9f586dd86b04fe984968b9de7f52209 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 1 leader_uuid: "af7bb239a5d6454daa140682a17b532d" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f9f586dd86b04fe984968b9de7f52209" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 36235 } } peers { permanent_uuid: "af7bb239a5d6454daa140682a17b532d" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 43367 } } peers { permanent_uuid: "180459af951d4aabbb5312ba1ae741bb" member_type: NON_VOTER last_known_addr { host: "127.0.57.193" port: 37035 } attrs { } } } W0507 06:28:30.787662 1705 tablet_service.cc:646] failed transaction from {username='slave'} at 127.0.0.1:52052: Illegal state: Replica af7bb239a5d6454daa140682a17b532d is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f9f586dd86b04fe984968b9de7f52209" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 36235 } } peers { permanent_uuid: "af7bb239a5d6454daa140682a17b532d" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 43367 } } peers { permanent_uuid: "180459af951d4aabbb5312ba1ae741bb" member_type: NON_VOTER last_known_addr { host: "127.0.57.193" port: 37035 } attrs { } } } W0507 06:28:30.788847 1706 tablet_service.cc:646] failed transaction from {username='slave'} at 127.0.0.1:52052: Illegal state: Replica af7bb239a5d6454daa140682a17b532d is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f9f586dd86b04fe984968b9de7f52209" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 36235 } } peers { permanent_uuid: "af7bb239a5d6454daa140682a17b532d" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 43367 } } peers { permanent_uuid: "180459af951d4aabbb5312ba1ae741bb" member_type: NON_VOTER last_known_addr { host: "127.0.57.193" port: 37035 } attrs { } } } W0507 06:28:30.789131 1705 tablet_service.cc:646] failed transaction from {username='slave'} at 127.0.0.1:52052: Illegal state: Replica af7bb239a5d6454daa140682a17b532d is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f9f586dd86b04fe984968b9de7f52209" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 36235 } } peers { permanent_uuid: "af7bb239a5d6454daa140682a17b532d" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 43367 } } peers { permanent_uuid: "180459af951d4aabbb5312ba1ae741bb" member_type: NON_VOTER last_known_addr { host: "127.0.57.193" port: 37035 } attrs { } } } W0507 06:28:30.792929 1705 tablet_service.cc:646] failed transaction from {username='slave'} at 127.0.0.1:52052: Illegal state: Replica af7bb239a5d6454daa140682a17b532d is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f9f586dd86b04fe984968b9de7f52209" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 36235 } } peers { permanent_uuid: "af7bb239a5d6454daa140682a17b532d" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 43367 } } peers { permanent_uuid: "180459af951d4aabbb5312ba1ae741bb" member_type: NON_VOTER last_known_addr { host: "127.0.57.193" port: 37035 } attrs { } } } I0507 06:28:30.830199 1974 raft_consensus.cc:472] T 9d26ac1dc0994a898fbb8aef9162ee61 P f9f586dd86b04fe984968b9de7f52209 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader af7bb239a5d6454daa140682a17b532d) I0507 06:28:30.830374 1974 raft_consensus.cc:494] T 9d26ac1dc0994a898fbb8aef9162ee61 P f9f586dd86b04fe984968b9de7f52209 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f9f586dd86b04fe984968b9de7f52209" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 36235 } } peers { permanent_uuid: "af7bb239a5d6454daa140682a17b532d" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 43367 } } peers { permanent_uuid: "180459af951d4aabbb5312ba1ae741bb" member_type: NON_VOTER last_known_addr { host: "127.0.57.193" port: 37035 } attrs { } } I0507 06:28:30.831459 1974 leader_election.cc:296] T 9d26ac1dc0994a898fbb8aef9162ee61 P f9f586dd86b04fe984968b9de7f52209 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers af7bb239a5d6454daa140682a17b532d (127.0.57.194:43367) W0507 06:28:30.835340 1827 tablet_service.cc:646] failed transaction from {username='slave'} at 127.0.0.1:58072: Illegal state: Replica f9f586dd86b04fe984968b9de7f52209 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 1 leader_uuid: "af7bb239a5d6454daa140682a17b532d" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f9f586dd86b04fe984968b9de7f52209" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 36235 } } peers { permanent_uuid: "af7bb239a5d6454daa140682a17b532d" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 43367 } } peers { permanent_uuid: "180459af951d4aabbb5312ba1ae741bb" member_type: NON_VOTER last_known_addr { host: "127.0.57.193" port: 37035 } attrs { } } } W0507 06:28:30.836323 1825 tablet_service.cc:646] failed transaction from {username='slave'} at 127.0.0.1:58072: Illegal state: Replica f9f586dd86b04fe984968b9de7f52209 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 1 leader_uuid: "af7bb239a5d6454daa140682a17b532d" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f9f586dd86b04fe984968b9de7f52209" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 36235 } } peers { permanent_uuid: "af7bb239a5d6454daa140682a17b532d" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 43367 } } peers { permanent_uuid: "180459af951d4aabbb5312ba1ae741bb" member_type: NON_VOTER last_known_addr { host: "127.0.57.193" port: 37035 } attrs { } } } W0507 06:28:30.837569 1825 tablet_service.cc:646] failed transaction from {username='slave'} at 127.0.0.1:58072: Illegal state: Replica f9f586dd86b04fe984968b9de7f52209 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 1 leader_uuid: "af7bb239a5d6454daa140682a17b532d" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f9f586dd86b04fe984968b9de7f52209" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 36235 } } peers { permanent_uuid: "af7bb239a5d6454daa140682a17b532d" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 43367 } } peers { permanent_uuid: "180459af951d4aabbb5312ba1ae741bb" member_type: NON_VOTER last_known_addr { host: "127.0.57.193" port: 37035 } attrs { } } } W0507 06:28:30.838809 1825 tablet_service.cc:646] failed transaction from {username='slave'} at 127.0.0.1:58072: Illegal state: Replica f9f586dd86b04fe984968b9de7f52209 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 1 leader_uuid: "af7bb239a5d6454daa140682a17b532d" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f9f586dd86b04fe984968b9de7f52209" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 36235 } } peers { permanent_uuid: "af7bb239a5d6454daa140682a17b532d" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 43367 } } peers { permanent_uuid: "180459af951d4aabbb5312ba1ae741bb" member_type: NON_VOTER last_known_addr { host: "127.0.57.193" port: 37035 } attrs { } } } I0507 06:28:30.842317 1746 tablet_service.cc:1286] Received RequestConsensusVote() RPC: tablet_id: "9d26ac1dc0994a898fbb8aef9162ee61" candidate_uuid: "f9f586dd86b04fe984968b9de7f52209" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "af7bb239a5d6454daa140682a17b532d" is_pre_election: true I0507 06:28:30.842607 1746 raft_consensus.cc:2346] T 9d26ac1dc0994a898fbb8aef9162ee61 P af7bb239a5d6454daa140682a17b532d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f9f586dd86b04fe984968b9de7f52209 in term 2. I0507 06:28:30.843104 1799 leader_election.cc:310] T 9d26ac1dc0994a898fbb8aef9162ee61 P f9f586dd86b04fe984968b9de7f52209 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: af7bb239a5d6454daa140682a17b532d, f9f586dd86b04fe984968b9de7f52209; no voters: I0507 06:28:30.843523 1974 raft_consensus.cc:2640] T 9d26ac1dc0994a898fbb8aef9162ee61 P f9f586dd86b04fe984968b9de7f52209 [term 1 FOLLOWER]: Leader pre-election won for term 2 I0507 06:28:30.843614 1974 raft_consensus.cc:472] T 9d26ac1dc0994a898fbb8aef9162ee61 P f9f586dd86b04fe984968b9de7f52209 [term 1 FOLLOWER]: Starting leader election (detected failure of leader af7bb239a5d6454daa140682a17b532d) I0507 06:28:30.843672 1974 raft_consensus.cc:2886] T 9d26ac1dc0994a898fbb8aef9162ee61 P f9f586dd86b04fe984968b9de7f52209 [term 1 FOLLOWER]: Advancing to term 2 I0507 06:28:30.845363 1974 raft_consensus.cc:494] T 9d26ac1dc0994a898fbb8aef9162ee61 P f9f586dd86b04fe984968b9de7f52209 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f9f586dd86b04fe984968b9de7f52209" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 36235 } } peers { permanent_uuid: "af7bb239a5d6454daa140682a17b532d" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 43367 } } peers { permanent_uuid: "180459af951d4aabbb5312ba1ae741bb" member_type: NON_VOTER last_known_addr { host: "127.0.57.193" port: 37035 } attrs { } } I0507 06:28:30.845752 1974 leader_election.cc:296] T 9d26ac1dc0994a898fbb8aef9162ee61 P f9f586dd86b04fe984968b9de7f52209 [CANDIDATE]: Term 2 election: Requested vote from peers af7bb239a5d6454daa140682a17b532d (127.0.57.194:43367) I0507 06:28:30.846201 1746 tablet_service.cc:1286] Received RequestConsensusVote() RPC: tablet_id: "9d26ac1dc0994a898fbb8aef9162ee61" candidate_uuid: "f9f586dd86b04fe984968b9de7f52209" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "af7bb239a5d6454daa140682a17b532d" I0507 06:28:30.847838 1746 raft_consensus.cc:2346] T 9d26ac1dc0994a898fbb8aef9162ee61 P af7bb239a5d6454daa140682a17b532d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f9f586dd86b04fe984968b9de7f52209 in term 2. I0507 06:28:30.848197 1799 leader_election.cc:310] T 9d26ac1dc0994a898fbb8aef9162ee61 P f9f586dd86b04fe984968b9de7f52209 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: af7bb239a5d6454daa140682a17b532d, f9f586dd86b04fe984968b9de7f52209; no voters: I0507 06:28:30.848448 1974 raft_consensus.cc:2640] T 9d26ac1dc0994a898fbb8aef9162ee61 P f9f586dd86b04fe984968b9de7f52209 [term 2 FOLLOWER]: Leader election won for term 2 I0507 06:28:30.848903 1974 raft_consensus.cc:667] T 9d26ac1dc0994a898fbb8aef9162ee61 P f9f586dd86b04fe984968b9de7f52209 [term 2 LEADER]: Becoming Leader. State: Replica: f9f586dd86b04fe984968b9de7f52209, State: Running, Role: LEADER I0507 06:28:30.849267 1974 consensus_queue.cc:206] T 9d26ac1dc0994a898fbb8aef9162ee61 P f9f586dd86b04fe984968b9de7f52209 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 10, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f9f586dd86b04fe984968b9de7f52209" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 36235 } } peers { permanent_uuid: "af7bb239a5d6454daa140682a17b532d" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 43367 } } peers { permanent_uuid: "180459af951d4aabbb5312ba1ae741bb" member_type: NON_VOTER last_known_addr { host: "127.0.57.193" port: 37035 } attrs { } } I0507 06:28:30.852696 1495 catalog_manager.cc:3988] T 9d26ac1dc0994a898fbb8aef9162ee61 P f9f586dd86b04fe984968b9de7f52209 reported cstate change: term changed from 1 to 2, leader changed from af7bb239a5d6454daa140682a17b532d (127.0.57.194) to f9f586dd86b04fe984968b9de7f52209 (127.0.57.195). New cstate: current_term: 2 leader_uuid: "f9f586dd86b04fe984968b9de7f52209" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f9f586dd86b04fe984968b9de7f52209" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 36235 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af7bb239a5d6454daa140682a17b532d" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 43367 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "180459af951d4aabbb5312ba1ae741bb" member_type: NON_VOTER last_known_addr { host: "127.0.57.193" port: 37035 } attrs { } health_report { overall_health: UNKNOWN } } } I0507 06:28:30.892681 1746 raft_consensus.cc:1181] T 9d26ac1dc0994a898fbb8aef9162ee61 P af7bb239a5d6454daa140682a17b532d [term 2 FOLLOWER]: Refusing update from remote peer f9f586dd86b04fe984968b9de7f52209: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 2 index: 12. (index mismatch) I0507 06:28:30.893549 1977 consensus_queue.cc:984] T 9d26ac1dc0994a898fbb8aef9162ee61 P f9f586dd86b04fe984968b9de7f52209 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af7bb239a5d6454daa140682a17b532d" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 43367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s I0507 06:28:30.897207 1623 raft_consensus.cc:2886] T 9d26ac1dc0994a898fbb8aef9162ee61 P 180459af951d4aabbb5312ba1ae741bb [term 1 LEARNER]: Advancing to term 2 I0507 06:28:30.899436 1623 raft_consensus.cc:1181] T 9d26ac1dc0994a898fbb8aef9162ee61 P 180459af951d4aabbb5312ba1ae741bb [term 2 LEARNER]: Refusing update from remote peer f9f586dd86b04fe984968b9de7f52209: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 2 index: 12. (index mismatch) I0507 06:28:30.900123 1977 consensus_queue.cc:984] T 9d26ac1dc0994a898fbb8aef9162ee61 P f9f586dd86b04fe984968b9de7f52209 [LEADER]: Connected to new peer: Peer: permanent_uuid: "180459af951d4aabbb5312ba1ae741bb" member_type: NON_VOTER last_known_addr { host: "127.0.57.193" port: 37035 } attrs { }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s Master Summary UUID | Address | Status ----------------------------------+--------------------+--------- 4365e3a555824c77be1797521bce6de6 | 127.0.57.254:38587 | HEALTHY Flag | Value | Tags | Master ---------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+------------------------- allow_unsafe_replication_factor | true | unsafe | all 1 server(s) checked catalog_manager_evict_excess_replicas | false | runtime,hidden | all 1 server(s) checked ipki_ca_key_size | 1024 | experimental | all 1 server(s) checked ipki_server_key_size | 1024 | experimental | all 1 server(s) checked never_fsync | true | unsafe,advanced | all 1 server(s) checked rpc_reuseport | true | experimental | all 1 server(s) checked rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked server_dump_info_format | pb | hidden | all 1 server(s) checked server_dump_info_path | /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.AddNonVoterReplica.1557210500371863-231/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked Tablet Server Summary UUID | Address | Status | Location ----------------------------------+--------------------+---------+---------- 180459af951d4aabbb5312ba1ae741bb | 127.0.57.193:37035 | HEALTHY | af7bb239a5d6454daa140682a17b532d | 127.0.57.194:43367 | HEALTHY | f9f586dd86b04fe984968b9de7f52209 | 127.0.57.195:36235 | HEALTHY | Tablet Server Location Summary Location | Count ----------+--------- | 3 Flag | Value | Tags | Tablet Server ---------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+------------------------- ipki_server_key_size | 1024 | experimental | all 3 server(s) checked local_ip_for_outbound_sockets | 127.0.57.193 | experimental | 127.0.57.193:37035 local_ip_for_outbound_sockets | 127.0.57.194 | experimental | 127.0.57.194:43367 local_ip_for_outbound_sockets | 127.0.57.195 | experimental | 127.0.57.195:36235 never_fsync | true | unsafe,advanced | all 3 server(s) checked rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked server_dump_info_format | pb | hidden | all 3 server(s) checked server_dump_info_path | /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.AddNonVoterReplica.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.0.57.193:37035 server_dump_info_path | /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.AddNonVoterReplica.1557210500371863-231/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.0.57.194:43367 server_dump_info_path | /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.AddNonVoterReplica.1557210500371863-231/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.0.57.195:36235 tablet_copy_download_file_inject_latency_ms | 1000 | hidden | all 3 server(s) checked Version Summary Version | Servers ------------------------+------------------------- 1.10.0-cdh6.x-SNAPSHOT | all 4 server(s) checked Tablet Summary Summary by table Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable -----------+----+---------+---------------+---------+------------+------------------+------------- TestTable | 2 | HEALTHY | 1 | 1 | 0 | 0 | 0 Tablet Replica Count Summary Statistic | Replica Count ----------------+--------------- Minimum | 1 First Quartile | 1 Median | 1 Third Quartile | 1 Maximum | 1 Total Count Summary | Total Count ----------------+------------- Masters | 1 Tablet Servers | 3 Tables | 1 Tablets | 1 Replicas | 3 ================== Warnings: ================== Some masters have unsafe, experimental, or hidden flags set Some tablet servers have unsafe, experimental, or hidden flags set OK I0507 06:28:31.048908 231 log_verifier.cc:121] Checking tablet 9d26ac1dc0994a898fbb8aef9162ee61 I0507 06:28:31.099504 231 log_verifier.cc:172] Verified matching terms for 23 ops in tablet 9d26ac1dc0994a898fbb8aef9162ee61 I0507 06:28:31.140365 231 external_mini_cluster.cc:1172] Attempting to check leaks for /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver pid 1536 I0507 06:28:31.141454 1663 generic_service.cc:171] Checking for leaks (request via RPC) I0507 06:28:31.220602 231 external_mini_cluster.cc:1131] Killing /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver with pid 1536 I0507 06:28:31.235618 231 external_mini_cluster.cc:1172] Attempting to check leaks for /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver pid 1670 I0507 06:28:31.236666 1786 generic_service.cc:171] Checking for leaks (request via RPC) I0507 06:28:31.325070 231 external_mini_cluster.cc:1131] Killing /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver with pid 1670 I0507 06:28:31.340912 231 external_mini_cluster.cc:1172] Attempting to check leaks for /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver pid 1792 I0507 06:28:31.342141 1908 generic_service.cc:171] Checking for leaks (request via RPC) I0507 06:28:31.422132 231 external_mini_cluster.cc:1131] Killing /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver with pid 1792 I0507 06:28:31.435392 231 external_mini_cluster.cc:1172] Attempting to check leaks for /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-master pid 1476 I0507 06:28:31.436426 1532 generic_service.cc:171] Checking for leaks (request via RPC) I0507 06:28:31.521024 231 external_mini_cluster.cc:1131] Killing /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-master with pid 1476 [ OK ] RaftConsensusNonVoterITest.AddNonVoterReplica (5014 ms) [ RUN ] RaftConsensusNonVoterITest.AddThenRemoveNonVoterReplica I0507 06:28:31.546193 231 test_util.cc:212] Using random seed: 1398904627 I0507 06:28:31.547832 231 ts_itest-base.cc:112] Starting cluster with: I0507 06:28:31.547880 231 ts_itest-base.cc:113] -------------- I0507 06:28:31.547894 231 ts_itest-base.cc:114] 3 tablet servers I0507 06:28:31.547916 231 ts_itest-base.cc:115] 2 replicas per TS I0507 06:28:31.547930 231 ts_itest-base.cc:116] -------------- I0507 06:28:31.549137 231 external_mini_cluster.cc:933] Running /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-master /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-master --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.AddThenRemoveNonVoterReplica.1557210500371863-231/raft_consensus-itest-cluster/master-0/wal --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.AddThenRemoveNonVoterReplica.1557210500371863-231/raft_consensus-itest-cluster/master-0/data --block_manager=log --webserver_interface=localhost --ipki_ca_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.0.57.254:42933 --webserver_interface=127.0.57.254 --webserver_port=0 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.AddThenRemoveNonVoterReplica.1557210500371863-231/raft_consensus-itest-cluster/master-0/logs --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.AddThenRemoveNonVoterReplica.1557210500371863-231/raft_consensus-itest-cluster/master-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --allow_unsafe_replication_factor=true --catalog_manager_evict_excess_replicas=false --rpc_reuseport=true with env {} WARNING: Logging before InitGoogleLogging() is written to STDERR W0507 06:28:31.803925 2041 flags.cc:404] Enabled unsafe flag: --allow_unsafe_replication_factor=true W0507 06:28:31.804160 2041 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0507 06:28:31.804219 2041 flags.cc:404] Enabled unsafe flag: --never_fsync=true W0507 06:28:31.806766 2041 flags.cc:404] Enabled experimental flag: --ipki_ca_key_size=1024 W0507 06:28:31.806828 2041 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W0507 06:28:31.806847 2041 flags.cc:404] Enabled experimental flag: --tsk_num_rsa_bits=512 W0507 06:28:31.806879 2041 flags.cc:404] Enabled experimental flag: --rpc_reuseport=true I0507 06:28:31.811504 2041 master_main.cc:73] Master server non-default flags: --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.AddThenRemoveNonVoterReplica.1557210500371863-231/raft_consensus-itest-cluster/master-0/data --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.AddThenRemoveNonVoterReplica.1557210500371863-231/raft_consensus-itest-cluster/master-0/wal --allow_unsafe_replication_factor=true --catalog_manager_evict_excess_replicas=false --ipki_ca_key_size=1024 --ipki_server_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.0.57.254:42933 --rpc_reuseport=true --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.AddThenRemoveNonVoterReplica.1557210500371863-231/raft_consensus-itest-cluster/master-0/data/info.pb --webserver_interface=127.0.57.254 --webserver_port=0 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.AddThenRemoveNonVoterReplica.1557210500371863-231/raft_consensus-itest-cluster/master-0/logs --logbuflevel=-1 --logtostderr=true Master server version: kudu 1.10.0-cdh6.x-SNAPSHOT revision 7d0ee26697926e87af9ef395008799577f6ab43f build type FASTDEBUG built by jenkins at 06 May 2019 22:44:21 PST on quasar-knmpfx-1.vpc.cloudera.com build id 322017 ASAN enabled I0507 06:28:31.818748 2041 system_ntp.cc:202] NTP initialized. Skew: 500ppm Current error: 30016us I0507 06:28:31.820021 2041 fs_manager.cc:263] Metadata directory not provided I0507 06:28:31.820091 2041 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0507 06:28:31.820348 2041 server_base.cc:433] Could not load existing FS layout: Not found: could not find a healthy instance file I0507 06:28:31.820417 2041 server_base.cc:434] Attempting to create new FS layout instead I0507 06:28:31.823505 2041 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.AddThenRemoveNonVoterReplica.1557210500371863-231/raft_consensus-itest-cluster/master-0/data/instance: uuid: "ca44e68a1ed248ddbdeb74e692af4e76" format_stamp: "Formatted at 2019-05-07 06:28:31 on dist-test-slave-dist-test-slave-rqp6" I0507 06:28:31.824126 2041 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.AddThenRemoveNonVoterReplica.1557210500371863-231/raft_consensus-itest-cluster/master-0/wal/instance: uuid: "ca44e68a1ed248ddbdeb74e692af4e76" format_stamp: "Formatted at 2019-05-07 06:28:31 on dist-test-slave-dist-test-slave-rqp6" I0507 06:28:31.831760 2041 fs_manager.cc:503] Time spent creating directory manager: real 0.007s user 0.009s sys 0.000s I0507 06:28:31.832918 2041 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0507 06:28:31.833534 2041 file_cache.cc:466] Constructed file cache lbm with capacity 419430 I0507 06:28:31.837141 2041 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.000s sys 0.000s I0507 06:28:31.837220 2041 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.AddThenRemoveNonVoterReplica.1557210500371863-231/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.AddThenRemoveNonVoterReplica.1557210500371863-231/raft_consensus-itest-cluster/master-0/wal uuid: "ca44e68a1ed248ddbdeb74e692af4e76" format_stamp: "Formatted at 2019-05-07 06:28:31 on dist-test-slave-dist-test-slave-rqp6" I0507 06:28:31.837355 2041 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.AddThenRemoveNonVoterReplica.1557210500371863-231/raft_consensus-itest-cluster/master-0/wal metadata directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.AddThenRemoveNonVoterReplica.1557210500371863-231/raft_consensus-itest-cluster/master-0/wal 1 data directories: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.AddThenRemoveNonVoterReplica.1557210500371863-231/raft_consensus-itest-cluster/master-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0507 06:28:31.863445 2041 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0507 06:28:31.879710 2041 rpc_server.cc:205] RPC server started. Bound to: 127.0.57.254:42933 I0507 06:28:31.881317 2041 webserver.cc:308] Webserver started at http://127.0.57.254:37781/ using document root and password file I0507 06:28:31.882318 2041 server_base.cc:608] Dumped server information to /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.AddThenRemoveNonVoterReplica.1557210500371863-231/raft_consensus-itest-cluster/master-0/data/info.pb I0507 06:28:31.885107 231 external_mini_cluster.cc:995] Started /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-master as pid 2041 I0507 06:28:31.886247 231 external_mini_cluster.cc:933] Running /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.AddThenRemoveNonVoterReplica.1557210500371863-231/raft_consensus-itest-cluster/ts-0/wal --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.AddThenRemoveNonVoterReplica.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data --block_manager=log --rpc_bind_addresses=127.0.57.193:0 --local_ip_for_outbound_sockets=127.0.57.193 --webserver_interface=127.0.57.193 --webserver_port=0 --tserver_master_addrs=127.0.57.254:42933 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.AddThenRemoveNonVoterReplica.1557210500371863-231/raft_consensus-itest-cluster/ts-0/logs --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.AddThenRemoveNonVoterReplica.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --log_cache_size_limit_mb=10 --consensus_rpc_timeout_ms=50 with env {} I0507 06:28:31.887313 2100 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0507 06:28:31.899971 2100 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P ca44e68a1ed248ddbdeb74e692af4e76: Bootstrap starting. I0507 06:28:31.902915 2100 tablet_bootstrap.cc:590] T 00000000000000000000000000000000 P ca44e68a1ed248ddbdeb74e692af4e76: No blocks or log segments found. Creating new log. I0507 06:28:31.903900 2100 log.cc:525] T 00000000000000000000000000000000 P ca44e68a1ed248ddbdeb74e692af4e76: Log is configured to *not* fsync() on all Append() calls I0507 06:28:31.905740 2100 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P ca44e68a1ed248ddbdeb74e692af4e76: No bootstrap required, opened a new log I0507 06:28:31.912647 2100 raft_consensus.cc:340] T 00000000000000000000000000000000 P ca44e68a1ed248ddbdeb74e692af4e76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca44e68a1ed248ddbdeb74e692af4e76" member_type: VOTER } I0507 06:28:31.912945 2100 raft_consensus.cc:366] T 00000000000000000000000000000000 P ca44e68a1ed248ddbdeb74e692af4e76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0507 06:28:31.913022 2100 raft_consensus.cc:705] T 00000000000000000000000000000000 P ca44e68a1ed248ddbdeb74e692af4e76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca44e68a1ed248ddbdeb74e692af4e76, State: Initialized, Role: FOLLOWER I0507 06:28:31.913561 2100 consensus_queue.cc:229] T 00000000000000000000000000000000 P ca44e68a1ed248ddbdeb74e692af4e76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca44e68a1ed248ddbdeb74e692af4e76" member_type: VOTER } I0507 06:28:31.913789 2100 raft_consensus.cc:380] T 00000000000000000000000000000000 P ca44e68a1ed248ddbdeb74e692af4e76 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0507 06:28:31.913888 2100 raft_consensus.cc:472] T 00000000000000000000000000000000 P ca44e68a1ed248ddbdeb74e692af4e76 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0507 06:28:31.913998 2100 raft_consensus.cc:2886] T 00000000000000000000000000000000 P ca44e68a1ed248ddbdeb74e692af4e76 [term 0 FOLLOWER]: Advancing to term 1 I0507 06:28:31.972371 2100 raft_consensus.cc:494] T 00000000000000000000000000000000 P ca44e68a1ed248ddbdeb74e692af4e76 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca44e68a1ed248ddbdeb74e692af4e76" member_type: VOTER } I0507 06:28:31.973140 2100 leader_election.cc:310] T 00000000000000000000000000000000 P ca44e68a1ed248ddbdeb74e692af4e76 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ca44e68a1ed248ddbdeb74e692af4e76; no voters: I0507 06:28:31.976948 2100 leader_election.cc:296] T 00000000000000000000000000000000 P ca44e68a1ed248ddbdeb74e692af4e76 [CANDIDATE]: Term 1 election: Requested vote from peers I0507 06:28:31.977227 2103 raft_consensus.cc:2640] T 00000000000000000000000000000000 P ca44e68a1ed248ddbdeb74e692af4e76 [term 1 FOLLOWER]: Leader election won for term 1 I0507 06:28:31.980762 2103 raft_consensus.cc:667] T 00000000000000000000000000000000 P ca44e68a1ed248ddbdeb74e692af4e76 [term 1 LEADER]: Becoming Leader. State: Replica: ca44e68a1ed248ddbdeb74e692af4e76, State: Running, Role: LEADER I0507 06:28:31.981402 2103 consensus_queue.cc:206] T 00000000000000000000000000000000 P ca44e68a1ed248ddbdeb74e692af4e76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca44e68a1ed248ddbdeb74e692af4e76" member_type: VOTER } I0507 06:28:31.985759 2100 sys_catalog.cc:337] T 00000000000000000000000000000000 P ca44e68a1ed248ddbdeb74e692af4e76 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 1 leader_uuid: "ca44e68a1ed248ddbdeb74e692af4e76" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca44e68a1ed248ddbdeb74e692af4e76" member_type: VOTER } } I0507 06:28:31.985777 2105 sys_catalog.cc:337] T 00000000000000000000000000000000 P ca44e68a1ed248ddbdeb74e692af4e76 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ca44e68a1ed248ddbdeb74e692af4e76" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca44e68a1ed248ddbdeb74e692af4e76" member_type: VOTER } } I0507 06:28:31.985496 2104 sys_catalog.cc:337] T 00000000000000000000000000000000 P ca44e68a1ed248ddbdeb74e692af4e76 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ca44e68a1ed248ddbdeb74e692af4e76. Latest consensus state: current_term: 1 leader_uuid: "ca44e68a1ed248ddbdeb74e692af4e76" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca44e68a1ed248ddbdeb74e692af4e76" member_type: VOTER } } I0507 06:28:31.986244 2105 sys_catalog.cc:340] T 00000000000000000000000000000000 P ca44e68a1ed248ddbdeb74e692af4e76 [sys.catalog]: This master's current role is: LEADER I0507 06:28:31.986310 2104 sys_catalog.cc:340] T 00000000000000000000000000000000 P ca44e68a1ed248ddbdeb74e692af4e76 [sys.catalog]: This master's current role is: LEADER I0507 06:28:31.986922 2100 sys_catalog.cc:340] T 00000000000000000000000000000000 P ca44e68a1ed248ddbdeb74e692af4e76 [sys.catalog]: This master's current role is: LEADER I0507 06:28:31.989367 2100 sys_catalog.cc:424] T 00000000000000000000000000000000 P ca44e68a1ed248ddbdeb74e692af4e76 [sys.catalog]: configured and running, proceeding with master startup. I0507 06:28:31.989765 2107 catalog_manager.cc:1055] Loading table and tablet metadata into memory... I0507 06:28:31.996430 2107 catalog_manager.cc:1066] Initializing Kudu internal certificate authority... I0507 06:28:32.031185 2107 catalog_manager.cc:950] Generated new certificate authority record I0507 06:28:32.033124 2107 catalog_manager.cc:1075] Loading token signing keys... I0507 06:28:32.045112 2107 catalog_manager.cc:4292] T 00000000000000000000000000000000 P ca44e68a1ed248ddbdeb74e692af4e76: Generated new TSK 0 WARNING: Logging before InitGoogleLogging() is written to STDERR W0507 06:28:32.087168 2101 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0507 06:28:32.087409 2101 flags.cc:404] Enabled unsafe flag: --never_fsync=true W0507 06:28:32.090087 2101 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W0507 06:28:32.090232 2101 flags.cc:404] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.57.193 I0507 06:28:32.099380 2101 tablet_server_main.cc:78] Tablet server non-default flags: --consensus_rpc_timeout_ms=50 --log_cache_size_limit_mb=10 --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.AddThenRemoveNonVoterReplica.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.AddThenRemoveNonVoterReplica.1557210500371863-231/raft_consensus-itest-cluster/ts-0/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.0.57.193:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.AddThenRemoveNonVoterReplica.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data/info.pb --webserver_interface=127.0.57.193 --webserver_port=0 --tserver_master_addrs=127.0.57.254:42933 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.0.57.193 --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.AddThenRemoveNonVoterReplica.1557210500371863-231/raft_consensus-itest-cluster/ts-0/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.10.0-cdh6.x-SNAPSHOT revision 7d0ee26697926e87af9ef395008799577f6ab43f build type FASTDEBUG built by jenkins at 06 May 2019 22:44:21 PST on quasar-knmpfx-1.vpc.cloudera.com build id 322017 ASAN enabled I0507 06:28:32.108778 2101 system_ntp.cc:202] NTP initialized. Skew: 500ppm Current error: 30516us I0507 06:28:32.109616 2101 fs_manager.cc:263] Metadata directory not provided I0507 06:28:32.109740 2101 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0507 06:28:32.110060 2101 server_base.cc:433] Could not load existing FS layout: Not found: could not find a healthy instance file I0507 06:28:32.110142 2101 server_base.cc:434] Attempting to create new FS layout instead I0507 06:28:32.113052 2101 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.AddThenRemoveNonVoterReplica.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data/instance: uuid: "115b8fbd574849d8af7218153486becb" format_stamp: "Formatted at 2019-05-07 06:28:32 on dist-test-slave-dist-test-slave-rqp6" I0507 06:28:32.113835 2101 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.AddThenRemoveNonVoterReplica.1557210500371863-231/raft_consensus-itest-cluster/ts-0/wal/instance: uuid: "115b8fbd574849d8af7218153486becb" format_stamp: "Formatted at 2019-05-07 06:28:32 on dist-test-slave-dist-test-slave-rqp6" I0507 06:28:32.135370 2101 fs_manager.cc:503] Time spent creating directory manager: real 0.021s user 0.006s sys 0.014s I0507 06:28:32.136848 2101 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0507 06:28:32.137677 2101 file_cache.cc:466] Constructed file cache lbm with capacity 419430 I0507 06:28:32.141110 2101 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.000s sys 0.000s I0507 06:28:32.141186 2101 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.AddThenRemoveNonVoterReplica.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.AddThenRemoveNonVoterReplica.1557210500371863-231/raft_consensus-itest-cluster/ts-0/wal uuid: "115b8fbd574849d8af7218153486becb" format_stamp: "Formatted at 2019-05-07 06:28:32 on dist-test-slave-dist-test-slave-rqp6" I0507 06:28:32.141506 2101 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.AddThenRemoveNonVoterReplica.1557210500371863-231/raft_consensus-itest-cluster/ts-0/wal metadata directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.AddThenRemoveNonVoterReplica.1557210500371863-231/raft_consensus-itest-cluster/ts-0/wal 1 data directories: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.AddThenRemoveNonVoterReplica.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0507 06:28:32.173636 2101 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0507 06:28:32.175580 2101 ts_tablet_manager.cc:356] Loaded tablet metadata (0 total tablets, 0 live tablets) I0507 06:28:32.175680 2101 ts_tablet_manager.cc:376] Registered 0 tablets I0507 06:28:32.335274 2101 rpc_server.cc:205] RPC server started. Bound to: 127.0.57.193:36949 I0507 06:28:32.336745 2101 webserver.cc:308] Webserver started at http://127.0.57.193:45381/ using document root and password file I0507 06:28:32.337476 2101 server_base.cc:608] Dumped server information to /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.AddThenRemoveNonVoterReplica.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data/info.pb I0507 06:28:32.338377 231 external_mini_cluster.cc:995] Started /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver as pid 2101 I0507 06:28:32.339069 231 external_mini_cluster.cc:933] Running /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.AddThenRemoveNonVoterReplica.1557210500371863-231/raft_consensus-itest-cluster/ts-1/wal --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.AddThenRemoveNonVoterReplica.1557210500371863-231/raft_consensus-itest-cluster/ts-1/data --block_manager=log --rpc_bind_addresses=127.0.57.194:0 --local_ip_for_outbound_sockets=127.0.57.194 --webserver_interface=127.0.57.194 --webserver_port=0 --tserver_master_addrs=127.0.57.254:42933 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.AddThenRemoveNonVoterReplica.1557210500371863-231/raft_consensus-itest-cluster/ts-1/logs --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.AddThenRemoveNonVoterReplica.1557210500371863-231/raft_consensus-itest-cluster/ts-1/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --log_cache_size_limit_mb=10 --consensus_rpc_timeout_ms=50 with env {} I0507 06:28:32.358294 2230 heartbeater.cc:345] Connected to a master server at 127.0.57.254:42933 I0507 06:28:32.358515 2230 heartbeater.cc:426] Registering TS with master... I0507 06:28:32.359102 2230 heartbeater.cc:475] Master 127.0.57.254:42933 requested a full tablet report, sending... I0507 06:28:32.360875 2067 ts_manager.cc:159] Registered new tserver with Master: 115b8fbd574849d8af7218153486becb (127.0.57.193:36949) I0507 06:28:32.362426 2067 master_service.cc:273] Signed X509 certificate for tserver {username='slave'} at 127.0.57.193:49407 WARNING: Logging before InitGoogleLogging() is written to STDERR W0507 06:28:32.510685 2234 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0507 06:28:32.510908 2234 flags.cc:404] Enabled unsafe flag: --never_fsync=true W0507 06:28:32.513204 2234 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W0507 06:28:32.513324 2234 flags.cc:404] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.57.194 I0507 06:28:32.517506 2234 tablet_server_main.cc:78] Tablet server non-default flags: --consensus_rpc_timeout_ms=50 --log_cache_size_limit_mb=10 --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.AddThenRemoveNonVoterReplica.1557210500371863-231/raft_consensus-itest-cluster/ts-1/data --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.AddThenRemoveNonVoterReplica.1557210500371863-231/raft_consensus-itest-cluster/ts-1/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.0.57.194:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.AddThenRemoveNonVoterReplica.1557210500371863-231/raft_consensus-itest-cluster/ts-1/data/info.pb --webserver_interface=127.0.57.194 --webserver_port=0 --tserver_master_addrs=127.0.57.254:42933 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.0.57.194 --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.AddThenRemoveNonVoterReplica.1557210500371863-231/raft_consensus-itest-cluster/ts-1/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.10.0-cdh6.x-SNAPSHOT revision 7d0ee26697926e87af9ef395008799577f6ab43f build type FASTDEBUG built by jenkins at 06 May 2019 22:44:21 PST on quasar-knmpfx-1.vpc.cloudera.com build id 322017 ASAN enabled I0507 06:28:32.525640 2234 system_ntp.cc:202] NTP initialized. Skew: 500ppm Current error: 30516us I0507 06:28:32.526362 2234 fs_manager.cc:263] Metadata directory not provided I0507 06:28:32.526445 2234 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0507 06:28:32.526803 2234 server_base.cc:433] Could not load existing FS layout: Not found: could not find a healthy instance file I0507 06:28:32.526885 2234 server_base.cc:434] Attempting to create new FS layout instead I0507 06:28:32.529386 2234 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.AddThenRemoveNonVoterReplica.1557210500371863-231/raft_consensus-itest-cluster/ts-1/data/instance: uuid: "c3974624d38d47c9a1e971a1facee6cd" format_stamp: "Formatted at 2019-05-07 06:28:32 on dist-test-slave-dist-test-slave-rqp6" I0507 06:28:32.529995 2234 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.AddThenRemoveNonVoterReplica.1557210500371863-231/raft_consensus-itest-cluster/ts-1/wal/instance: uuid: "c3974624d38d47c9a1e971a1facee6cd" format_stamp: "Formatted at 2019-05-07 06:28:32 on dist-test-slave-dist-test-slave-rqp6" I0507 06:28:32.537065 2234 fs_manager.cc:503] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s I0507 06:28:32.538153 2234 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0507 06:28:32.538975 2234 file_cache.cc:466] Constructed file cache lbm with capacity 419430 I0507 06:28:32.542140 2234 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s I0507 06:28:32.542217 2234 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.AddThenRemoveNonVoterReplica.1557210500371863-231/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.AddThenRemoveNonVoterReplica.1557210500371863-231/raft_consensus-itest-cluster/ts-1/wal uuid: "c3974624d38d47c9a1e971a1facee6cd" format_stamp: "Formatted at 2019-05-07 06:28:32 on dist-test-slave-dist-test-slave-rqp6" I0507 06:28:32.542457 2234 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.AddThenRemoveNonVoterReplica.1557210500371863-231/raft_consensus-itest-cluster/ts-1/wal metadata directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.AddThenRemoveNonVoterReplica.1557210500371863-231/raft_consensus-itest-cluster/ts-1/wal 1 data directories: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.AddThenRemoveNonVoterReplica.1557210500371863-231/raft_consensus-itest-cluster/ts-1/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0507 06:28:32.595372 2234 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0507 06:28:32.597532 2234 ts_tablet_manager.cc:356] Loaded tablet metadata (0 total tablets, 0 live tablets) I0507 06:28:32.597599 2234 ts_tablet_manager.cc:376] Registered 0 tablets I0507 06:28:32.624502 2234 rpc_server.cc:205] RPC server started. Bound to: 127.0.57.194:42979 I0507 06:28:32.625926 2234 webserver.cc:308] Webserver started at http://127.0.57.194:43619/ using document root and password file I0507 06:28:32.626664 2234 server_base.cc:608] Dumped server information to /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.AddThenRemoveNonVoterReplica.1557210500371863-231/raft_consensus-itest-cluster/ts-1/data/info.pb I0507 06:28:32.627548 231 external_mini_cluster.cc:995] Started /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver as pid 2234 I0507 06:28:32.628311 231 external_mini_cluster.cc:933] Running /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.AddThenRemoveNonVoterReplica.1557210500371863-231/raft_consensus-itest-cluster/ts-2/wal --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.AddThenRemoveNonVoterReplica.1557210500371863-231/raft_consensus-itest-cluster/ts-2/data --block_manager=log --rpc_bind_addresses=127.0.57.195:0 --local_ip_for_outbound_sockets=127.0.57.195 --webserver_interface=127.0.57.195 --webserver_port=0 --tserver_master_addrs=127.0.57.254:42933 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.AddThenRemoveNonVoterReplica.1557210500371863-231/raft_consensus-itest-cluster/ts-2/logs --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.AddThenRemoveNonVoterReplica.1557210500371863-231/raft_consensus-itest-cluster/ts-2/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --log_cache_size_limit_mb=10 --consensus_rpc_timeout_ms=50 with env {} I0507 06:28:32.648831 2353 heartbeater.cc:345] Connected to a master server at 127.0.57.254:42933 I0507 06:28:32.649077 2353 heartbeater.cc:426] Registering TS with master... I0507 06:28:32.649713 2353 heartbeater.cc:475] Master 127.0.57.254:42933 requested a full tablet report, sending... I0507 06:28:32.651002 2067 ts_manager.cc:159] Registered new tserver with Master: c3974624d38d47c9a1e971a1facee6cd (127.0.57.194:42979) I0507 06:28:32.652109 2067 master_service.cc:273] Signed X509 certificate for tserver {username='slave'} at 127.0.57.194:35889 WARNING: Logging before InitGoogleLogging() is written to STDERR W0507 06:28:32.864035 2356 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0507 06:28:32.864331 2356 flags.cc:404] Enabled unsafe flag: --never_fsync=true W0507 06:28:32.867677 2356 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W0507 06:28:32.867863 2356 flags.cc:404] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.57.195 I0507 06:28:32.873991 2356 tablet_server_main.cc:78] Tablet server non-default flags: --consensus_rpc_timeout_ms=50 --log_cache_size_limit_mb=10 --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.AddThenRemoveNonVoterReplica.1557210500371863-231/raft_consensus-itest-cluster/ts-2/data --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.AddThenRemoveNonVoterReplica.1557210500371863-231/raft_consensus-itest-cluster/ts-2/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.0.57.195:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.AddThenRemoveNonVoterReplica.1557210500371863-231/raft_consensus-itest-cluster/ts-2/data/info.pb --webserver_interface=127.0.57.195 --webserver_port=0 --tserver_master_addrs=127.0.57.254:42933 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.0.57.195 --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.AddThenRemoveNonVoterReplica.1557210500371863-231/raft_consensus-itest-cluster/ts-2/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.10.0-cdh6.x-SNAPSHOT revision 7d0ee26697926e87af9ef395008799577f6ab43f build type FASTDEBUG built by jenkins at 06 May 2019 22:44:21 PST on quasar-knmpfx-1.vpc.cloudera.com build id 322017 ASAN enabled I0507 06:28:32.882534 2356 system_ntp.cc:202] NTP initialized. Skew: 500ppm Current error: 30516us I0507 06:28:32.883472 2356 fs_manager.cc:263] Metadata directory not provided I0507 06:28:32.883566 2356 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0507 06:28:32.883946 2356 server_base.cc:433] Could not load existing FS layout: Not found: could not find a healthy instance file I0507 06:28:32.884053 2356 server_base.cc:434] Attempting to create new FS layout instead I0507 06:28:32.886767 2356 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.AddThenRemoveNonVoterReplica.1557210500371863-231/raft_consensus-itest-cluster/ts-2/data/instance: uuid: "d83fc0be0e694b7d8ee02426d3a553f1" format_stamp: "Formatted at 2019-05-07 06:28:32 on dist-test-slave-dist-test-slave-rqp6" I0507 06:28:32.887353 2356 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.AddThenRemoveNonVoterReplica.1557210500371863-231/raft_consensus-itest-cluster/ts-2/wal/instance: uuid: "d83fc0be0e694b7d8ee02426d3a553f1" format_stamp: "Formatted at 2019-05-07 06:28:32 on dist-test-slave-dist-test-slave-rqp6" I0507 06:28:32.894399 2356 fs_manager.cc:503] Time spent creating directory manager: real 0.007s user 0.007s sys 0.002s I0507 06:28:32.895514 2356 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0507 06:28:32.896139 2356 file_cache.cc:466] Constructed file cache lbm with capacity 419430 I0507 06:28:32.899747 2356 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s I0507 06:28:32.899832 2356 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.AddThenRemoveNonVoterReplica.1557210500371863-231/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.AddThenRemoveNonVoterReplica.1557210500371863-231/raft_consensus-itest-cluster/ts-2/wal uuid: "d83fc0be0e694b7d8ee02426d3a553f1" format_stamp: "Formatted at 2019-05-07 06:28:32 on dist-test-slave-dist-test-slave-rqp6" I0507 06:28:32.899989 2356 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.AddThenRemoveNonVoterReplica.1557210500371863-231/raft_consensus-itest-cluster/ts-2/wal metadata directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.AddThenRemoveNonVoterReplica.1557210500371863-231/raft_consensus-itest-cluster/ts-2/wal 1 data directories: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.AddThenRemoveNonVoterReplica.1557210500371863-231/raft_consensus-itest-cluster/ts-2/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0507 06:28:32.936401 2356 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0507 06:28:32.938396 2356 ts_tablet_manager.cc:356] Loaded tablet metadata (0 total tablets, 0 live tablets) I0507 06:28:32.938455 2356 ts_tablet_manager.cc:376] Registered 0 tablets I0507 06:28:32.971490 2356 rpc_server.cc:205] RPC server started. Bound to: 127.0.57.195:33903 I0507 06:28:32.973079 2356 webserver.cc:308] Webserver started at http://127.0.57.195:45443/ using document root and password file I0507 06:28:32.973937 2356 server_base.cc:608] Dumped server information to /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.AddThenRemoveNonVoterReplica.1557210500371863-231/raft_consensus-itest-cluster/ts-2/data/info.pb I0507 06:28:32.980787 231 external_mini_cluster.cc:995] Started /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver as pid 2356 I0507 06:28:32.991847 2475 heartbeater.cc:345] Connected to a master server at 127.0.57.254:42933 I0507 06:28:32.992087 2475 heartbeater.cc:426] Registering TS with master... I0507 06:28:32.992825 2475 heartbeater.cc:475] Master 127.0.57.254:42933 requested a full tablet report, sending... I0507 06:28:32.994246 2067 ts_manager.cc:159] Registered new tserver with Master: d83fc0be0e694b7d8ee02426d3a553f1 (127.0.57.195:33903) I0507 06:28:32.995260 2067 master_service.cc:273] Signed X509 certificate for tserver {username='slave'} at 127.0.57.195:40389 I0507 06:28:32.996407 231 external_mini_cluster.cc:623] 3 TS(s) registered with all masters I0507 06:28:33.010196 2066 catalog_manager.cc:1415] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53358: name: "TestTable" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 2 split_rows_range_bounds { } partition_schema { range_schema { columns { name: "key" } } } I0507 06:28:33.042754 2167 tablet_service.cc:1016] Processing CreateTablet for tablet d58928e7866a4149a437597f8bfcb2ac (table=TestTable [id=f35e96e810a445ac9906e7c985846e91]), partition=RANGE (key) PARTITION UNBOUNDED I0507 06:28:33.042754 2412 tablet_service.cc:1016] Processing CreateTablet for tablet d58928e7866a4149a437597f8bfcb2ac (table=TestTable [id=f35e96e810a445ac9906e7c985846e91]), partition=RANGE (key) PARTITION UNBOUNDED I0507 06:28:33.043721 2412 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet d58928e7866a4149a437597f8bfcb2ac. 1 dirs total, 0 dirs full, 0 dirs failed I0507 06:28:33.043725 2167 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet d58928e7866a4149a437597f8bfcb2ac. 1 dirs total, 0 dirs full, 0 dirs failed I0507 06:28:33.055517 2492 tablet_bootstrap.cc:439] T d58928e7866a4149a437597f8bfcb2ac P d83fc0be0e694b7d8ee02426d3a553f1: Bootstrap starting. I0507 06:28:33.057046 2493 tablet_bootstrap.cc:439] T d58928e7866a4149a437597f8bfcb2ac P 115b8fbd574849d8af7218153486becb: Bootstrap starting. I0507 06:28:33.059406 2493 tablet_bootstrap.cc:590] T d58928e7866a4149a437597f8bfcb2ac P 115b8fbd574849d8af7218153486becb: No blocks or log segments found. Creating new log. I0507 06:28:33.060389 2493 log.cc:525] T d58928e7866a4149a437597f8bfcb2ac P 115b8fbd574849d8af7218153486becb: Log is configured to *not* fsync() on all Append() calls I0507 06:28:33.062136 2492 tablet_bootstrap.cc:590] T d58928e7866a4149a437597f8bfcb2ac P d83fc0be0e694b7d8ee02426d3a553f1: No blocks or log segments found. Creating new log. I0507 06:28:33.063966 2492 log.cc:525] T d58928e7866a4149a437597f8bfcb2ac P d83fc0be0e694b7d8ee02426d3a553f1: Log is configured to *not* fsync() on all Append() calls I0507 06:28:33.066112 2493 tablet_bootstrap.cc:439] T d58928e7866a4149a437597f8bfcb2ac P 115b8fbd574849d8af7218153486becb: No bootstrap required, opened a new log I0507 06:28:33.066391 2493 ts_tablet_manager.cc:1053] T d58928e7866a4149a437597f8bfcb2ac P 115b8fbd574849d8af7218153486becb: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.000s I0507 06:28:33.066424 2492 tablet_bootstrap.cc:439] T d58928e7866a4149a437597f8bfcb2ac P d83fc0be0e694b7d8ee02426d3a553f1: No bootstrap required, opened a new log I0507 06:28:33.066692 2492 ts_tablet_manager.cc:1053] T d58928e7866a4149a437597f8bfcb2ac P d83fc0be0e694b7d8ee02426d3a553f1: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.004s I0507 06:28:33.072775 2493 raft_consensus.cc:340] T d58928e7866a4149a437597f8bfcb2ac P 115b8fbd574849d8af7218153486becb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "115b8fbd574849d8af7218153486becb" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 36949 } } peers { permanent_uuid: "d83fc0be0e694b7d8ee02426d3a553f1" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 33903 } } I0507 06:28:33.072774 2492 raft_consensus.cc:340] T d58928e7866a4149a437597f8bfcb2ac P d83fc0be0e694b7d8ee02426d3a553f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "115b8fbd574849d8af7218153486becb" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 36949 } } peers { permanent_uuid: "d83fc0be0e694b7d8ee02426d3a553f1" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 33903 } } I0507 06:28:33.073107 2492 raft_consensus.cc:366] T d58928e7866a4149a437597f8bfcb2ac P d83fc0be0e694b7d8ee02426d3a553f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0507 06:28:33.073112 2493 raft_consensus.cc:366] T d58928e7866a4149a437597f8bfcb2ac P 115b8fbd574849d8af7218153486becb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0507 06:28:33.073155 2492 raft_consensus.cc:705] T d58928e7866a4149a437597f8bfcb2ac P d83fc0be0e694b7d8ee02426d3a553f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d83fc0be0e694b7d8ee02426d3a553f1, State: Initialized, Role: FOLLOWER I0507 06:28:33.073194 2493 raft_consensus.cc:705] T d58928e7866a4149a437597f8bfcb2ac P 115b8fbd574849d8af7218153486becb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 115b8fbd574849d8af7218153486becb, State: Initialized, Role: FOLLOWER I0507 06:28:33.073657 2492 consensus_queue.cc:229] T d58928e7866a4149a437597f8bfcb2ac P d83fc0be0e694b7d8ee02426d3a553f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "115b8fbd574849d8af7218153486becb" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 36949 } } peers { permanent_uuid: "d83fc0be0e694b7d8ee02426d3a553f1" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 33903 } } I0507 06:28:33.073653 2493 consensus_queue.cc:229] T d58928e7866a4149a437597f8bfcb2ac P 115b8fbd574849d8af7218153486becb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "115b8fbd574849d8af7218153486becb" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 36949 } } peers { permanent_uuid: "d83fc0be0e694b7d8ee02426d3a553f1" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 33903 } } I0507 06:28:33.074455 2475 heartbeater.cc:467] Master 127.0.57.254:42933 was elected leader, sending a full tablet report... I0507 06:28:33.074748 2230 heartbeater.cc:467] Master 127.0.57.254:42933 was elected leader, sending a full tablet report... I0507 06:28:33.074970 2492 ts_tablet_manager.cc:1081] T d58928e7866a4149a437597f8bfcb2ac P d83fc0be0e694b7d8ee02426d3a553f1: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s I0507 06:28:33.075768 2493 ts_tablet_manager.cc:1081] T d58928e7866a4149a437597f8bfcb2ac P 115b8fbd574849d8af7218153486becb: Time spent starting tablet: real 0.009s user 0.003s sys 0.006s W0507 06:28:33.089861 2231 tablet.cc:1786] T d58928e7866a4149a437597f8bfcb2ac P 115b8fbd574849d8af7218153486becb: Can't schedule compaction. Clean time has not been advanced past its initial value. I0507 06:28:33.125357 2497 raft_consensus.cc:472] T d58928e7866a4149a437597f8bfcb2ac P 115b8fbd574849d8af7218153486becb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0507 06:28:33.125629 2497 raft_consensus.cc:494] T d58928e7866a4149a437597f8bfcb2ac P 115b8fbd574849d8af7218153486becb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "115b8fbd574849d8af7218153486becb" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 36949 } } peers { permanent_uuid: "d83fc0be0e694b7d8ee02426d3a553f1" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 33903 } } I0507 06:28:33.126976 2497 leader_election.cc:296] T d58928e7866a4149a437597f8bfcb2ac P 115b8fbd574849d8af7218153486becb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d83fc0be0e694b7d8ee02426d3a553f1 (127.0.57.195:33903) I0507 06:28:33.133838 2432 tablet_service.cc:1286] Received RequestConsensusVote() RPC: tablet_id: "d58928e7866a4149a437597f8bfcb2ac" candidate_uuid: "115b8fbd574849d8af7218153486becb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d83fc0be0e694b7d8ee02426d3a553f1" is_pre_election: true I0507 06:28:33.134203 2432 raft_consensus.cc:2346] T d58928e7866a4149a437597f8bfcb2ac P d83fc0be0e694b7d8ee02426d3a553f1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 115b8fbd574849d8af7218153486becb in term 0. I0507 06:28:33.134943 2118 leader_election.cc:310] T d58928e7866a4149a437597f8bfcb2ac P 115b8fbd574849d8af7218153486becb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 115b8fbd574849d8af7218153486becb, d83fc0be0e694b7d8ee02426d3a553f1; no voters: I0507 06:28:33.135414 2497 raft_consensus.cc:2640] T d58928e7866a4149a437597f8bfcb2ac P 115b8fbd574849d8af7218153486becb [term 0 FOLLOWER]: Leader pre-election won for term 1 I0507 06:28:33.135512 2497 raft_consensus.cc:472] T d58928e7866a4149a437597f8bfcb2ac P 115b8fbd574849d8af7218153486becb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0507 06:28:33.135598 2497 raft_consensus.cc:2886] T d58928e7866a4149a437597f8bfcb2ac P 115b8fbd574849d8af7218153486becb [term 0 FOLLOWER]: Advancing to term 1 I0507 06:28:33.164357 2497 raft_consensus.cc:494] T d58928e7866a4149a437597f8bfcb2ac P 115b8fbd574849d8af7218153486becb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "115b8fbd574849d8af7218153486becb" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 36949 } } peers { permanent_uuid: "d83fc0be0e694b7d8ee02426d3a553f1" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 33903 } } I0507 06:28:33.165087 2497 leader_election.cc:296] T d58928e7866a4149a437597f8bfcb2ac P 115b8fbd574849d8af7218153486becb [CANDIDATE]: Term 1 election: Requested vote from peers d83fc0be0e694b7d8ee02426d3a553f1 (127.0.57.195:33903) I0507 06:28:33.165881 2432 tablet_service.cc:1286] Received RequestConsensusVote() RPC: tablet_id: "d58928e7866a4149a437597f8bfcb2ac" candidate_uuid: "115b8fbd574849d8af7218153486becb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d83fc0be0e694b7d8ee02426d3a553f1" I0507 06:28:33.166051 2432 raft_consensus.cc:2886] T d58928e7866a4149a437597f8bfcb2ac P d83fc0be0e694b7d8ee02426d3a553f1 [term 0 FOLLOWER]: Advancing to term 1 I0507 06:28:33.168048 2432 raft_consensus.cc:2346] T d58928e7866a4149a437597f8bfcb2ac P d83fc0be0e694b7d8ee02426d3a553f1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 115b8fbd574849d8af7218153486becb in term 1. I0507 06:28:33.168622 2118 leader_election.cc:310] T d58928e7866a4149a437597f8bfcb2ac P 115b8fbd574849d8af7218153486becb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 115b8fbd574849d8af7218153486becb, d83fc0be0e694b7d8ee02426d3a553f1; no voters: I0507 06:28:33.168942 2497 raft_consensus.cc:2640] T d58928e7866a4149a437597f8bfcb2ac P 115b8fbd574849d8af7218153486becb [term 1 FOLLOWER]: Leader election won for term 1 I0507 06:28:33.169484 2497 raft_consensus.cc:667] T d58928e7866a4149a437597f8bfcb2ac P 115b8fbd574849d8af7218153486becb [term 1 LEADER]: Becoming Leader. State: Replica: 115b8fbd574849d8af7218153486becb, State: Running, Role: LEADER I0507 06:28:33.169919 2497 consensus_queue.cc:206] T d58928e7866a4149a437597f8bfcb2ac P 115b8fbd574849d8af7218153486becb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "115b8fbd574849d8af7218153486becb" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 36949 } } peers { permanent_uuid: "d83fc0be0e694b7d8ee02426d3a553f1" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 33903 } } I0507 06:28:33.175343 2066 catalog_manager.cc:3988] T d58928e7866a4149a437597f8bfcb2ac P 115b8fbd574849d8af7218153486becb reported cstate change: term changed from 0 to 1, leader changed from to 115b8fbd574849d8af7218153486becb (127.0.57.193). New cstate: current_term: 1 leader_uuid: "115b8fbd574849d8af7218153486becb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "115b8fbd574849d8af7218153486becb" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 36949 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d83fc0be0e694b7d8ee02426d3a553f1" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 33903 } health_report { overall_health: UNKNOWN } } } I0507 06:28:33.191646 231 external_mini_cluster.cc:623] 3 TS(s) registered with all masters I0507 06:28:33.193374 231 ts_itest-base.cc:232] Waiting for 1 tablets on tserver 115b8fbd574849d8af7218153486becb to finish bootstrapping I0507 06:28:33.200702 231 ts_itest-base.cc:232] Waiting for 0 tablets on tserver c3974624d38d47c9a1e971a1facee6cd to finish bootstrapping I0507 06:28:33.208897 231 ts_itest-base.cc:232] Waiting for 1 tablets on tserver d83fc0be0e694b7d8ee02426d3a553f1 to finish bootstrapping W0507 06:28:33.226321 2476 tablet.cc:1786] T d58928e7866a4149a437597f8bfcb2ac P d83fc0be0e694b7d8ee02426d3a553f1: Can't schedule compaction. Clean time has not been advanced past its initial value. I0507 06:28:33.571164 2432 raft_consensus.cc:1181] T d58928e7866a4149a437597f8bfcb2ac P d83fc0be0e694b7d8ee02426d3a553f1 [term 1 FOLLOWER]: Refusing update from remote peer 115b8fbd574849d8af7218153486becb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch) I0507 06:28:33.572413 2497 consensus_queue.cc:984] T d58928e7866a4149a437597f8bfcb2ac P 115b8fbd574849d8af7218153486becb [LEADER]: Connected to new peer: Peer: permanent_uuid: "d83fc0be0e694b7d8ee02426d3a553f1" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 33903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0507 06:28:33.589211 2187 consensus_queue.cc:206] T d58928e7866a4149a437597f8bfcb2ac P 115b8fbd574849d8af7218153486becb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "115b8fbd574849d8af7218153486becb" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 36949 } } peers { permanent_uuid: "d83fc0be0e694b7d8ee02426d3a553f1" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 33903 } } peers { permanent_uuid: "c3974624d38d47c9a1e971a1facee6cd" member_type: NON_VOTER last_known_addr { host: "127.0.57.194" port: 42979 } attrs { } } I0507 06:28:33.591624 2432 raft_consensus.cc:1181] T d58928e7866a4149a437597f8bfcb2ac P d83fc0be0e694b7d8ee02426d3a553f1 [term 1 FOLLOWER]: Refusing update from remote peer 115b8fbd574849d8af7218153486becb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0507 06:28:33.592242 2499 consensus_queue.cc:984] T d58928e7866a4149a437597f8bfcb2ac P 115b8fbd574849d8af7218153486becb [LEADER]: Connected to new peer: Peer: permanent_uuid: "d83fc0be0e694b7d8ee02426d3a553f1" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 33903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s I0507 06:28:33.595219 2499 raft_consensus.cc:2788] T d58928e7866a4149a437597f8bfcb2ac P 115b8fbd574849d8af7218153486becb [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER c3974624d38d47c9a1e971a1facee6cd (127.0.57.194) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "115b8fbd574849d8af7218153486becb" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 36949 } } peers { permanent_uuid: "d83fc0be0e694b7d8ee02426d3a553f1" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 33903 } } peers { permanent_uuid: "c3974624d38d47c9a1e971a1facee6cd" member_type: NON_VOTER last_known_addr { host: "127.0.57.194" port: 42979 } attrs { } } } I0507 06:28:33.595654 2432 raft_consensus.cc:2788] T d58928e7866a4149a437597f8bfcb2ac P d83fc0be0e694b7d8ee02426d3a553f1 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER c3974624d38d47c9a1e971a1facee6cd (127.0.57.194) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "115b8fbd574849d8af7218153486becb" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 36949 } } peers { permanent_uuid: "d83fc0be0e694b7d8ee02426d3a553f1" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 33903 } } peers { permanent_uuid: "c3974624d38d47c9a1e971a1facee6cd" member_type: NON_VOTER last_known_addr { host: "127.0.57.194" port: 42979 } attrs { } } } W0507 06:28:33.598700 2121 consensus_peers.cc:457] T d58928e7866a4149a437597f8bfcb2ac P 115b8fbd574849d8af7218153486becb -> Peer c3974624d38d47c9a1e971a1facee6cd (127.0.57.194:42979): Couldn't send request to peer c3974624d38d47c9a1e971a1facee6cd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d58928e7866a4149a437597f8bfcb2ac. This is attempt 1: this message will repeat every 5th retry. W0507 06:28:33.646371 2118 consensus_peers.cc:457] T d58928e7866a4149a437597f8bfcb2ac P 115b8fbd574849d8af7218153486becb -> Peer d83fc0be0e694b7d8ee02426d3a553f1 (127.0.57.195:33903): Couldn't send request to peer d83fc0be0e694b7d8ee02426d3a553f1. Status: Timed out: UpdateConsensus RPC to 127.0.57.195:33903 timed out after 0.050s (SENT). This is attempt 1: this message will repeat every 5th retry. I0507 06:28:33.654645 2353 heartbeater.cc:467] Master 127.0.57.254:42933 was elected leader, sending a full tablet report... W0507 06:28:33.661891 2432 rpcz_store.cc:253] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.57.193:38719 (request call id 8) took 66 ms (66 ms). Client timeout 50 ms (50 ms) W0507 06:28:33.663266 2432 rpcz_store.cc:259] Trace: 0507 06:28:33.595377 (+ 0us) service_pool.cc:162] Inserting onto call queue 0507 06:28:33.595415 (+ 38us) service_pool.cc:221] Handling call 0507 06:28:33.595510 (+ 95us) raft_consensus.cc:1391] Updating replica for 0 ops 0507 06:28:33.595596 (+ 86us) raft_consensus.cc:1454] Early marking committed up to index 2 0507 06:28:33.661467 (+ 65871us) log.cc:588] Serialized 14 byte log entry 0507 06:28:33.661644 (+ 177us) raft_consensus.cc:1459] Triggering prepare for 0 ops 0507 06:28:33.661677 (+ 33us) raft_consensus.cc:1565] Marking committed up to 2 0507 06:28:33.661692 (+ 15us) raft_consensus.cc:1614] Filling consensus response to leader. 0507 06:28:33.661714 (+ 22us) raft_consensus.cc:1608] UpdateReplicas() finished 0507 06:28:33.661789 (+ 75us) inbound_call.cc:162] Queueing success response Metrics: {"raft.queue_time_us":77,"raft.run_cpu_time_us":47,"raft.run_wall_time_us":48} I0507 06:28:33.663291 2066 catalog_manager.cc:3988] T d58928e7866a4149a437597f8bfcb2ac P d83fc0be0e694b7d8ee02426d3a553f1 reported cstate change: config changed from index -1 to 2, NON_VOTER c3974624d38d47c9a1e971a1facee6cd (127.0.57.194) added. New cstate: current_term: 1 leader_uuid: "115b8fbd574849d8af7218153486becb" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "115b8fbd574849d8af7218153486becb" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 36949 } } peers { permanent_uuid: "d83fc0be0e694b7d8ee02426d3a553f1" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 33903 } } peers { permanent_uuid: "c3974624d38d47c9a1e971a1facee6cd" member_type: NON_VOTER last_known_addr { host: "127.0.57.194" port: 42979 } attrs { } } } I0507 06:28:33.664007 231 test_util.cc:212] Using random seed: 1401022448 I0507 06:28:33.679498 231 test_workload.cc:306] TestWorkload: Skipping table creation because table TestTable already exists I0507 06:28:34.200068 2520 ts_tablet_manager.cc:676] T d58928e7866a4149a437597f8bfcb2ac P c3974624d38d47c9a1e971a1facee6cd: Initiating tablet copy from peer 115b8fbd574849d8af7218153486becb (127.0.57.193:36949) I0507 06:28:34.200426 2520 tablet_copy_client.cc:240] T d58928e7866a4149a437597f8bfcb2ac P c3974624d38d47c9a1e971a1facee6cd: tablet copy: Beginning tablet copy session from remote peer at address 127.0.57.193:36949 I0507 06:28:34.208956 2207 tablet_copy_service.cc:135] P 115b8fbd574849d8af7218153486becb: Received BeginTabletCopySession request for tablet d58928e7866a4149a437597f8bfcb2ac from peer c3974624d38d47c9a1e971a1facee6cd ({username='slave'} at 127.0.57.194:51581) I0507 06:28:34.209172 2207 tablet_copy_service.cc:156] P 115b8fbd574849d8af7218153486becb: Beginning new tablet copy session on tablet d58928e7866a4149a437597f8bfcb2ac from peer c3974624d38d47c9a1e971a1facee6cd at {username='slave'} at 127.0.57.194:51581: session id = c3974624d38d47c9a1e971a1facee6cd-d58928e7866a4149a437597f8bfcb2ac I0507 06:28:34.215453 2207 tablet_copy_source_session.cc:202] T d58928e7866a4149a437597f8bfcb2ac P 115b8fbd574849d8af7218153486becb: Tablet Copy: opened 0 blocks and 1 log segments I0507 06:28:34.219835 2520 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet d58928e7866a4149a437597f8bfcb2ac. 1 dirs total, 0 dirs full, 0 dirs failed W0507 06:28:34.232229 2520 rpcz_store.cc:253] Call kudu.consensus.ConsensusService.StartTabletCopy from 127.0.57.193:55071 (request call id 2) took 38 ms (38 ms). Client timeout 50 ms (50 ms) W0507 06:28:34.233237 2520 rpcz_store.cc:259] Trace: 0507 06:28:34.193570 (+ 0us) service_pool.cc:162] Inserting onto call queue 0507 06:28:34.193699 (+ 129us) service_pool.cc:221] Handling call 0507 06:28:34.200134 (+ 6435us) ts_tablet_manager.cc:677] T d58928e7866a4149a437597f8bfcb2ac P c3974624d38d47c9a1e971a1facee6cd: Initiating tablet copy from peer 115b8fbd574849d8af7218153486becb (127.0.57.193:36949) 0507 06:28:34.217318 (+ 17184us) tablet_copy_client.cc:259] Tablet copy session begun 0507 06:28:34.227720 (+ 10402us) tablet_metadata.cc:548] Metadata flushed 0507 06:28:34.228078 (+ 358us) tablet_copy_client.cc:364] Wrote new tablet metadata 0507 06:28:34.231373 (+ 3295us) tablet_copy_client.cc:376] Wrote new consensus metadata 0507 06:28:34.231791 (+ 418us) tablet_replica.cc:145] Creating consensus instance 0507 06:28:34.232147 (+ 356us) ts_tablet_manager.cc:732] Replica registered. 0507 06:28:34.232186 (+ 39us) inbound_call.cc:162] Queueing success response Metrics: {"dns_us":57,"tablet-copy.queue_time_us":5818,"thread_start_us":6560,"threads_started":1} I0507 06:28:34.233625 2520 tablet_copy_client.cc:575] T d58928e7866a4149a437597f8bfcb2ac P c3974624d38d47c9a1e971a1facee6cd: tablet copy: Starting download of 0 data blocks... I0507 06:28:34.233883 2520 tablet_copy_client.cc:538] T d58928e7866a4149a437597f8bfcb2ac P c3974624d38d47c9a1e971a1facee6cd: tablet copy: Starting download of 1 WAL segments... I0507 06:28:34.235903 2520 tablet_copy_client.cc:414] T d58928e7866a4149a437597f8bfcb2ac P c3974624d38d47c9a1e971a1facee6cd: tablet copy: Tablet Copy complete. Replacing tablet superblock. I0507 06:28:34.263533 2520 tablet_bootstrap.cc:439] T d58928e7866a4149a437597f8bfcb2ac P c3974624d38d47c9a1e971a1facee6cd: Bootstrap starting. I0507 06:28:34.270126 2520 log.cc:525] T d58928e7866a4149a437597f8bfcb2ac P c3974624d38d47c9a1e971a1facee6cd: Log is configured to *not* fsync() on all Append() calls I0507 06:28:34.274353 2520 tablet_bootstrap.cc:439] T d58928e7866a4149a437597f8bfcb2ac P c3974624d38d47c9a1e971a1facee6cd: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 4 replicates I0507 06:28:34.274965 2520 tablet_bootstrap.cc:439] T d58928e7866a4149a437597f8bfcb2ac P c3974624d38d47c9a1e971a1facee6cd: Bootstrap complete. I0507 06:28:34.275290 2520 ts_tablet_manager.cc:1053] T d58928e7866a4149a437597f8bfcb2ac P c3974624d38d47c9a1e971a1facee6cd: Time spent bootstrapping tablet: real 0.012s user 0.008s sys 0.002s I0507 06:28:34.281124 2520 raft_consensus.cc:340] T d58928e7866a4149a437597f8bfcb2ac P c3974624d38d47c9a1e971a1facee6cd [term 1 LEARNER]: Replica starting. Triggering 4 pending transactions. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "115b8fbd574849d8af7218153486becb" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 36949 } } peers { permanent_uuid: "d83fc0be0e694b7d8ee02426d3a553f1" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 33903 } } peers { permanent_uuid: "c3974624d38d47c9a1e971a1facee6cd" member_type: NON_VOTER last_known_addr { host: "127.0.57.194" port: 42979 } attrs { } } I0507 06:28:34.285495 2520 raft_consensus.cc:705] T d58928e7866a4149a437597f8bfcb2ac P c3974624d38d47c9a1e971a1facee6cd [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c3974624d38d47c9a1e971a1facee6cd, State: Initialized, Role: LEARNER I0507 06:28:34.285934 2520 consensus_queue.cc:229] T d58928e7866a4149a437597f8bfcb2ac P c3974624d38d47c9a1e971a1facee6cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "115b8fbd574849d8af7218153486becb" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 36949 } } peers { permanent_uuid: "d83fc0be0e694b7d8ee02426d3a553f1" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 33903 } } peers { permanent_uuid: "c3974624d38d47c9a1e971a1facee6cd" member_type: NON_VOTER last_known_addr { host: "127.0.57.194" port: 42979 } attrs { } } I0507 06:28:34.289589 2520 ts_tablet_manager.cc:1081] T d58928e7866a4149a437597f8bfcb2ac P c3974624d38d47c9a1e971a1facee6cd: Time spent starting tablet: real 0.014s user 0.016s sys 0.000s I0507 06:28:34.290611 2207 tablet_copy_service.cc:325] P 115b8fbd574849d8af7218153486becb: Request end of tablet copy session c3974624d38d47c9a1e971a1facee6cd-d58928e7866a4149a437597f8bfcb2ac received from {username='slave'} at 127.0.57.194:51581 I0507 06:28:34.290747 2207 tablet_copy_service.cc:413] P 115b8fbd574849d8af7218153486becb: ending tablet copy session c3974624d38d47c9a1e971a1facee6cd-d58928e7866a4149a437597f8bfcb2ac on tablet d58928e7866a4149a437597f8bfcb2ac with peer c3974624d38d47c9a1e971a1facee6cd Master Summary UUID | Address | Status ----------------------------------+--------------------+--------- ca44e68a1ed248ddbdeb74e692af4e76 | 127.0.57.254:42933 | HEALTHY Flag | Value | Tags | Master ---------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+------------------------- allow_unsafe_replication_factor | true | unsafe | all 1 server(s) checked catalog_manager_evict_excess_replicas | false | runtime,hidden | all 1 server(s) checked ipki_ca_key_size | 1024 | experimental | all 1 server(s) checked ipki_server_key_size | 1024 | experimental | all 1 server(s) checked never_fsync | true | unsafe,advanced | all 1 server(s) checked rpc_reuseport | true | experimental | all 1 server(s) checked rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked server_dump_info_format | pb | hidden | all 1 server(s) checked server_dump_info_path | /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.AddThenRemoveNonVoterReplica.1557210500371863-231/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked Tablet Server Summary UUID | Address | Status | Location ----------------------------------+--------------------+---------+---------- 115b8fbd574849d8af7218153486becb | 127.0.57.193:36949 | HEALTHY | c3974624d38d47c9a1e971a1facee6cd | 127.0.57.194:42979 | HEALTHY | d83fc0be0e694b7d8ee02426d3a553f1 | 127.0.57.195:33903 | HEALTHY | Tablet Server Location Summary Location | Count ----------+--------- | 3 Flag | Value | Tags | Tablet Server ----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+------------------------- ipki_server_key_size | 1024 | experimental | all 3 server(s) checked local_ip_for_outbound_sockets | 127.0.57.193 | experimental | 127.0.57.193:36949 local_ip_for_outbound_sockets | 127.0.57.194 | experimental | 127.0.57.194:42979 local_ip_for_outbound_sockets | 127.0.57.195 | experimental | 127.0.57.195:33903 never_fsync | true | unsafe,advanced | all 3 server(s) checked rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked server_dump_info_format | pb | hidden | all 3 server(s) checked server_dump_info_path | /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.AddThenRemoveNonVoterReplica.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.0.57.193:36949 server_dump_info_path | /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.AddThenRemoveNonVoterReplica.1557210500371863-231/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.0.57.194:42979 server_dump_info_path | /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.AddThenRemoveNonVoterReplica.1557210500371863-231/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.0.57.195:33903 Version Summary Version | Servers ------------------------+------------------------- 1.10.0-cdh6.x-SNAPSHOT | all 4 server(s) checked Tablet Summary Tablet d58928e7866a4149a437597f8bfcb2ac of table 'TestTable' is conflicted: 1 replicas' active configs disagree with the leader master's 115b8fbd574849d8af7218153486becb (127.0.57.193:36949): RUNNING [LEADER] d83fc0be0e694b7d8ee02426d3a553f1 (127.0.57.195:33903): RUNNING c3974624d38d47c9a1e971a1facee6cd (127.0.57.194:42979): RUNNING [NONVOTER] All reported replicas are: A = 115b8fbd574849d8af7218153486becb B = d83fc0be0e694b7d8ee02426d3a553f1 C = c3974624d38d47c9a1e971a1facee6cd The consensus matrix is: Config source | Replicas | Current term | Config index | Committed? ---------------+--------------+--------------+--------------+------------ master | A* B C~ | | | Yes A | A* B C~ | 1 | 2 | Yes B | A* B C~ | 1 | 2 | Yes C | A B C~ | 1 | 2 | Yes Summary by table Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable -----------+----+--------------------+---------------+---------+------------+------------------+------------- TestTable | 2 | CONSENSUS_MISMATCH | 1 | 0 | 0 | 0 | 1 Tablet Replica Count Summary Statistic | Replica Count ----------------+--------------- Minimum | 1 First Quartile | 1 Median | 1 Third Quartile | 1 Maximum | 1 Total Count Summary | Total Count ----------------+------------- Masters | 1 Tablet Servers | 3 Tables | 1 Tablets | 1 Replicas | 3 ================== Warnings: ================== Some masters have unsafe, experimental, or hidden flags set Some tablet servers have unsafe, experimental, or hidden flags set ================== Errors: ================== Corruption: table consistency check error: 1 out of 1 table(s) are not healthy FAILED I0507 06:28:34.378609 231 cluster_verifier.cc:82] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors Master Summary UUID | Address | Status ----------------------------------+--------------------+--------- ca44e68a1ed248ddbdeb74e692af4e76 | 127.0.57.254:42933 | HEALTHY Flag | Value | Tags | Master ---------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+------------------------- allow_unsafe_replication_factor | true | unsafe | all 1 server(s) checked catalog_manager_evict_excess_replicas | false | runtime,hidden | all 1 server(s) checked ipki_ca_key_size | 1024 | experimental | all 1 server(s) checked ipki_server_key_size | 1024 | experimental | all 1 server(s) checked never_fsync | true | unsafe,advanced | all 1 server(s) checked rpc_reuseport | true | experimental | all 1 server(s) checked rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked server_dump_info_format | pb | hidden | all 1 server(s) checked server_dump_info_path | /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.AddThenRemoveNonVoterReplica.1557210500371863-231/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked Tablet Server Summary UUID | Address | Status | Location ----------------------------------+--------------------+---------+---------- 115b8fbd574849d8af7218153486becb | 127.0.57.193:36949 | HEALTHY | c3974624d38d47c9a1e971a1facee6cd | 127.0.57.194:42979 | HEALTHY | d83fc0be0e694b7d8ee02426d3a553f1 | 127.0.57.195:33903 | HEALTHY | Tablet Server Location Summary Location | Count ----------+--------- | 3 Flag | Value | Tags | Tablet Server ----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+------------------------- ipki_server_key_size | 1024 | experimental | all 3 server(s) checked local_ip_for_outbound_sockets | 127.0.57.193 | experimental | 127.0.57.193:36949 local_ip_for_outbound_sockets | 127.0.57.194 | experimental | 127.0.57.194:42979 local_ip_for_outbound_sockets | 127.0.57.195 | experimental | 127.0.57.195:33903 never_fsync | true | unsafe,advanced | all 3 server(s) checked rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked server_dump_info_format | pb | hidden | all 3 server(s) checked server_dump_info_path | /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.AddThenRemoveNonVoterReplica.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.0.57.193:36949 server_dump_info_path | /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.AddThenRemoveNonVoterReplica.1557210500371863-231/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.0.57.194:42979 server_dump_info_path | /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.AddThenRemoveNonVoterReplica.1557210500371863-231/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.0.57.195:33903 Version Summary Version | Servers ------------------------+------------------------- 1.10.0-cdh6.x-SNAPSHOT | all 4 server(s) checked Tablet Summary Tablet d58928e7866a4149a437597f8bfcb2ac of table 'TestTable' is conflicted: 1 replicas' active configs disagree with the leader master's 115b8fbd574849d8af7218153486becb (127.0.57.193:36949): RUNNING [LEADER] d83fc0be0e694b7d8ee02426d3a553f1 (127.0.57.195:33903): RUNNING c3974624d38d47c9a1e971a1facee6cd (127.0.57.194:42979): RUNNING [NONVOTER] All reported replicas are: A = 115b8fbd574849d8af7218153486becb B = d83fc0be0e694b7d8ee02426d3a553f1 C = c3974624d38d47c9a1e971a1facee6cd The consensus matrix is: Config source | Replicas | Current term | Config index | Committed? ---------------+--------------+--------------+--------------+------------ master | A* B C~ | | | Yes A | A* B C~ | 1 | 2 | Yes B | A* B C~ | 1 | 2 | Yes C | A B C~ | 1 | 2 | Yes Summary by table Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable -----------+----+--------------------+---------------+---------+------------+------------------+------------- TestTable | 2 | CONSENSUS_MISMATCH | 1 | 0 | 0 | 0 | 1 Tablet Replica Count Summary Statistic | Replica Count ----------------+--------------- Minimum | 1 First Quartile | 1 Median | 1 Third Quartile | 1 Maximum | 1 Total Count Summary | Total Count ----------------+------------- Masters | 1 Tablet Servers | 3 Tables | 1 Tablets | 1 Replicas | 3 ================== Warnings: ================== Some masters have unsafe, experimental, or hidden flags set Some tablet servers have unsafe, experimental, or hidden flags set ================== Errors: ================== Corruption: table consistency check error: 1 out of 1 table(s) are not healthy FAILED I0507 06:28:34.593855 231 cluster_verifier.cc:82] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors I0507 06:28:34.792716 2310 raft_consensus.cc:1126] T d58928e7866a4149a437597f8bfcb2ac P c3974624d38d47c9a1e971a1facee6cd [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.8] Dedup: 1.6->[1.7-1.8] Master Summary UUID | Address | Status ----------------------------------+--------------------+--------- ca44e68a1ed248ddbdeb74e692af4e76 | 127.0.57.254:42933 | HEALTHY Flag | Value | Tags | Master ---------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+------------------------- allow_unsafe_replication_factor | true | unsafe | all 1 server(s) checked catalog_manager_evict_excess_replicas | false | runtime,hidden | all 1 server(s) checked ipki_ca_key_size | 1024 | experimental | all 1 server(s) checked ipki_server_key_size | 1024 | experimental | all 1 server(s) checked never_fsync | true | unsafe,advanced | all 1 server(s) checked rpc_reuseport | true | experimental | all 1 server(s) checked rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked server_dump_info_format | pb | hidden | all 1 server(s) checked server_dump_info_path | /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.AddThenRemoveNonVoterReplica.1557210500371863-231/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked Tablet Server Summary UUID | Address | Status | Location ----------------------------------+--------------------+---------+---------- 115b8fbd574849d8af7218153486becb | 127.0.57.193:36949 | HEALTHY | c3974624d38d47c9a1e971a1facee6cd | 127.0.57.194:42979 | HEALTHY | d83fc0be0e694b7d8ee02426d3a553f1 | 127.0.57.195:33903 | HEALTHY | Tablet Server Location Summary Location | Count ----------+--------- | 3 Flag | Value | Tags | Tablet Server ----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+------------------------- ipki_server_key_size | 1024 | experimental | all 3 server(s) checked local_ip_for_outbound_sockets | 127.0.57.193 | experimental | 127.0.57.193:36949 local_ip_for_outbound_sockets | 127.0.57.194 | experimental | 127.0.57.194:42979 local_ip_for_outbound_sockets | 127.0.57.195 | experimental | 127.0.57.195:33903 never_fsync | true | unsafe,advanced | all 3 server(s) checked rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked server_dump_info_format | pb | hidden | all 3 server(s) checked server_dump_info_path | /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.AddThenRemoveNonVoterReplica.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.0.57.193:36949 server_dump_info_path | /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.AddThenRemoveNonVoterReplica.1557210500371863-231/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.0.57.194:42979 server_dump_info_path | /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.AddThenRemoveNonVoterReplica.1557210500371863-231/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.0.57.195:33903 Version Summary Version | Servers ------------------------+------------------------- 1.10.0-cdh6.x-SNAPSHOT | all 4 server(s) checked Tablet Summary Summary by table Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable -----------+----+---------+---------------+---------+------------+------------------+------------- TestTable | 2 | HEALTHY | 1 | 1 | 0 | 0 | 0 Tablet Replica Count Summary Statistic | Replica Count ----------------+--------------- Minimum | 1 First Quartile | 1 Median | 1 Third Quartile | 1 Maximum | 1 Total Count Summary | Total Count ----------------+------------- Masters | 1 Tablet Servers | 3 Tables | 1 Tablets | 1 Replicas | 3 ================== Warnings: ================== Some masters have unsafe, experimental, or hidden flags set Some tablet servers have unsafe, experimental, or hidden flags set OK I0507 06:28:34.890100 231 log_verifier.cc:121] Checking tablet d58928e7866a4149a437597f8bfcb2ac I0507 06:28:34.917871 231 log_verifier.cc:172] Verified matching terms for 8 ops in tablet d58928e7866a4149a437597f8bfcb2ac I0507 06:28:35.415422 2187 consensus_queue.cc:206] T d58928e7866a4149a437597f8bfcb2ac P 115b8fbd574849d8af7218153486becb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "115b8fbd574849d8af7218153486becb" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 36949 } } peers { permanent_uuid: "d83fc0be0e694b7d8ee02426d3a553f1" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 33903 } } I0507 06:28:35.422049 2432 raft_consensus.cc:1181] T d58928e7866a4149a437597f8bfcb2ac P d83fc0be0e694b7d8ee02426d3a553f1 [term 1 FOLLOWER]: Refusing update from remote peer 115b8fbd574849d8af7218153486becb: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch) I0507 06:28:35.422782 2584 consensus_queue.cc:984] T d58928e7866a4149a437597f8bfcb2ac P 115b8fbd574849d8af7218153486becb [LEADER]: Connected to new peer: Peer: permanent_uuid: "d83fc0be0e694b7d8ee02426d3a553f1" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 33903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s I0507 06:28:35.431128 2616 raft_consensus.cc:2788] T d58928e7866a4149a437597f8bfcb2ac P 115b8fbd574849d8af7218153486becb [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 2 to 9, NON_VOTER c3974624d38d47c9a1e971a1facee6cd (127.0.57.194) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "115b8fbd574849d8af7218153486becb" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 36949 } } peers { permanent_uuid: "d83fc0be0e694b7d8ee02426d3a553f1" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 33903 } } } I0507 06:28:35.433660 2432 raft_consensus.cc:2788] T d58928e7866a4149a437597f8bfcb2ac P d83fc0be0e694b7d8ee02426d3a553f1 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 2 to 9, NON_VOTER c3974624d38d47c9a1e971a1facee6cd (127.0.57.194) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "115b8fbd574849d8af7218153486becb" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 36949 } } peers { permanent_uuid: "d83fc0be0e694b7d8ee02426d3a553f1" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 33903 } } } W0507 06:28:35.483757 2118 consensus_peers.cc:457] T d58928e7866a4149a437597f8bfcb2ac P 115b8fbd574849d8af7218153486becb -> Peer d83fc0be0e694b7d8ee02426d3a553f1 (127.0.57.195:33903): Couldn't send request to peer d83fc0be0e694b7d8ee02426d3a553f1. Status: Timed out: UpdateConsensus RPC to 127.0.57.195:33903 timed out after 0.050s (SENT). This is attempt 1: this message will repeat every 5th retry. I0507 06:28:35.599227 2066 catalog_manager.cc:3988] T d58928e7866a4149a437597f8bfcb2ac P 115b8fbd574849d8af7218153486becb reported cstate change: config changed from index 2 to 9, NON_VOTER c3974624d38d47c9a1e971a1facee6cd (127.0.57.194) evicted. New cstate: current_term: 1 leader_uuid: "115b8fbd574849d8af7218153486becb" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "115b8fbd574849d8af7218153486becb" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 36949 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d83fc0be0e694b7d8ee02426d3a553f1" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 33903 } health_report { overall_health: HEALTHY } } } W0507 06:28:35.607215 2432 rpcz_store.cc:253] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.57.193:38719 (request call id 19) took 173 ms (173 ms). Client timeout 50 ms (50 ms) W0507 06:28:35.607707 2432 rpcz_store.cc:259] Trace: 0507 06:28:35.433260 (+ 0us) service_pool.cc:162] Inserting onto call queue 0507 06:28:35.433380 (+ 120us) service_pool.cc:221] Handling call 0507 06:28:35.433530 (+ 150us) raft_consensus.cc:1391] Updating replica for 0 ops 0507 06:28:35.433599 (+ 69us) raft_consensus.cc:1454] Early marking committed up to index 9 0507 06:28:35.592557 (+158958us) log.cc:588] Serialized 14 byte log entry 0507 06:28:35.606899 (+ 14342us) raft_consensus.cc:1459] Triggering prepare for 0 ops 0507 06:28:35.606936 (+ 37us) raft_consensus.cc:1565] Marking committed up to 9 0507 06:28:35.606962 (+ 26us) raft_consensus.cc:1614] Filling consensus response to leader. 0507 06:28:35.607002 (+ 40us) raft_consensus.cc:1608] UpdateReplicas() finished 0507 06:28:35.607089 (+ 87us) inbound_call.cc:162] Queueing success response Metrics: {"raft.queue_time_us":14309,"thread_start_us":14122,"threads_started":1} I0507 06:28:35.612376 2066 catalog_manager.cc:3390] Sending DeleteTablet(TABLET_DATA_TOMBSTONED) for tablet d58928e7866a4149a437597f8bfcb2ac on c3974624d38d47c9a1e971a1facee6cd (127.0.57.194:42979) (TS c3974624d38d47c9a1e971a1facee6cd not found in new config with opid_index 9) I0507 06:28:35.621287 2290 tablet_service.cc:1057] Processing DeleteTablet for tablet d58928e7866a4149a437597f8bfcb2ac with delete_type TABLET_DATA_TOMBSTONED (TS c3974624d38d47c9a1e971a1facee6cd not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:33688 I0507 06:28:35.622201 2625 tablet_replica.cc:264] T d58928e7866a4149a437597f8bfcb2ac P c3974624d38d47c9a1e971a1facee6cd: stopping tablet replica I0507 06:28:35.622655 2625 raft_consensus.cc:2143] T d58928e7866a4149a437597f8bfcb2ac P c3974624d38d47c9a1e971a1facee6cd [term 1 LEARNER]: Raft consensus shutting down. I0507 06:28:35.623018 2625 raft_consensus.cc:2170] T d58928e7866a4149a437597f8bfcb2ac P c3974624d38d47c9a1e971a1facee6cd [term 1 LEARNER]: Raft consensus is shut down! I0507 06:28:35.625787 2625 ts_tablet_manager.cc:1377] T d58928e7866a4149a437597f8bfcb2ac P c3974624d38d47c9a1e971a1facee6cd: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED Master Summary UUID | Address | Status ----------------------------------+--------------------+--------- ca44e68a1ed248ddbdeb74e692af4e76 | 127.0.57.254:42933 | HEALTHY Flag | Value | Tags | Master ---------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+------------------------- allow_unsafe_replication_factor | true | unsafe | all 1 server(s) checked catalog_manager_evict_excess_replicas | false | runtime,hidden | all 1 server(s) checked ipki_ca_key_size | 1024 | experimental | all 1 server(s) checked ipki_server_key_size | 1024 | experimental | all 1 server(s) checked never_fsync | true | unsafe,advanced | all 1 server(s) checked rpc_reuseport | true | experimental | all 1 server(s) checked rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked server_dump_info_format | pb | hidden | all 1 server(s) checked server_dump_info_path | /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.AddThenRemoveNonVoterReplica.1557210500371863-231/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked Tablet Server Summary UUID | Address | Status | Location ----------------------------------+--------------------+---------+---------- 115b8fbd574849d8af7218153486becb | 127.0.57.193:36949 | HEALTHY | c3974624d38d47c9a1e971a1facee6cd | 127.0.57.194:42979 | HEALTHY | d83fc0be0e694b7d8ee02426d3a553f1 | 127.0.57.195:33903 | HEALTHY | Tablet Server Location Summary Location | Count ----------+--------- | 3 Flag | Value | Tags | Tablet Server ----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+------------------------- ipki_server_key_size | 1024 | experimental | all 3 server(s) checked local_ip_for_outbound_sockets | 127.0.57.193 | experimental | 127.0.57.193:36949 local_ip_for_outbound_sockets | 127.0.57.194 | experimental | 127.0.57.194:42979 local_ip_for_outbound_sockets | 127.0.57.195 | experimental | 127.0.57.195:33903 never_fsync | true | unsafe,advanced | all 3 server(s) checked rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked server_dump_info_format | pb | hidden | all 3 server(s) checked server_dump_info_path | /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.AddThenRemoveNonVoterReplica.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.0.57.193:36949 server_dump_info_path | /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.AddThenRemoveNonVoterReplica.1557210500371863-231/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.0.57.194:42979 server_dump_info_path | /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.AddThenRemoveNonVoterReplica.1557210500371863-231/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.0.57.195:33903 Version Summary Version | Servers ------------------------+------------------------- 1.10.0-cdh6.x-SNAPSHOT | all 4 server(s) checked Tablet Summary Summary by table Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable -----------+----+---------+---------------+---------+------------+------------------+------------- TestTable | 2 | HEALTHY | 1 | 1 | 0 | 0 | 0 Tablet Replica Count Summary Statistic | Replica Count ----------------+--------------- Minimum | 0 First Quartile | 0 Median | 1 Third Quartile | 1 Maximum | 1 Total Count Summary | Total Count ----------------+------------- Masters | 1 Tablet Servers | 3 Tables | 1 Tablets | 1 Replicas | 2 ================== Warnings: ================== Some masters have unsafe, experimental, or hidden flags set Some tablet servers have unsafe, experimental, or hidden flags set OK I0507 06:28:36.895926 231 log_verifier.cc:121] Checking tablet d58928e7866a4149a437597f8bfcb2ac I0507 06:28:36.898380 2625 ts_tablet_manager.cc:1390] T d58928e7866a4149a437597f8bfcb2ac P c3974624d38d47c9a1e971a1facee6cd: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8 I0507 06:28:36.898521 2625 log.cc:990] T d58928e7866a4149a437597f8bfcb2ac P c3974624d38d47c9a1e971a1facee6cd: Deleting WAL directory at /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.AddThenRemoveNonVoterReplica.1557210500371863-231/raft_consensus-itest-cluster/ts-1/wal/wals/d58928e7866a4149a437597f8bfcb2ac I0507 06:28:36.916314 2625 rpcz_store.cc:269] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33688 (request call id 1) took 1295ms. Request Metrics: {"tablet-delete.queue_time_us":417,"thread_start_us":274,"threads_started":1} I0507 06:28:36.917208 2051 catalog_manager.cc:3368] TS c3974624d38d47c9a1e971a1facee6cd (127.0.57.194:42979): tablet d58928e7866a4149a437597f8bfcb2ac (table TestTable [id=f35e96e810a445ac9906e7c985846e91]) successfully deleted I0507 06:28:36.928660 231 log_verifier.cc:172] Verified matching terms for 19 ops in tablet d58928e7866a4149a437597f8bfcb2ac I0507 06:28:37.036623 231 external_mini_cluster.cc:1172] Attempting to check leaks for /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver pid 2101 I0507 06:28:37.037600 2227 generic_service.cc:171] Checking for leaks (request via RPC) I0507 06:28:37.104218 231 external_mini_cluster.cc:1131] Killing /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver with pid 2101 I0507 06:28:37.116705 231 external_mini_cluster.cc:1172] Attempting to check leaks for /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver pid 2234 I0507 06:28:37.117846 2350 generic_service.cc:171] Checking for leaks (request via RPC) I0507 06:28:37.173008 231 external_mini_cluster.cc:1131] Killing /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver with pid 2234 I0507 06:28:37.184500 231 external_mini_cluster.cc:1172] Attempting to check leaks for /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver pid 2356 I0507 06:28:37.185731 2472 generic_service.cc:171] Checking for leaks (request via RPC) I0507 06:28:37.264427 231 external_mini_cluster.cc:1131] Killing /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver with pid 2356 I0507 06:28:37.279817 231 external_mini_cluster.cc:1172] Attempting to check leaks for /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-master pid 2041 I0507 06:28:37.280937 2097 generic_service.cc:171] Checking for leaks (request via RPC) I0507 06:28:37.362134 231 external_mini_cluster.cc:1131] Killing /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-master with pid 2041 [ OK ] RaftConsensusNonVoterITest.AddThenRemoveNonVoterReplica (5840 ms) [ RUN ] RaftConsensusNonVoterITest.NonVoterReplicasDoNotVote I0507 06:28:37.386598 231 test_util.cc:212] Using random seed: 1404745044 I0507 06:28:37.388300 231 ts_itest-base.cc:112] Starting cluster with: I0507 06:28:37.388337 231 ts_itest-base.cc:113] -------------- I0507 06:28:37.388368 231 ts_itest-base.cc:114] 3 tablet servers I0507 06:28:37.388387 231 ts_itest-base.cc:115] 2 replicas per TS I0507 06:28:37.388417 231 ts_itest-base.cc:116] -------------- I0507 06:28:37.389678 231 external_mini_cluster.cc:933] Running /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-master /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-master --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.NonVoterReplicasDoNotVote.1557210500371863-231/raft_consensus-itest-cluster/master-0/wal --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.NonVoterReplicasDoNotVote.1557210500371863-231/raft_consensus-itest-cluster/master-0/data --block_manager=log --webserver_interface=localhost --ipki_ca_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.0.57.254:46409 --webserver_interface=127.0.57.254 --webserver_port=0 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.NonVoterReplicasDoNotVote.1557210500371863-231/raft_consensus-itest-cluster/master-0/logs --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.NonVoterReplicasDoNotVote.1557210500371863-231/raft_consensus-itest-cluster/master-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --allow_unsafe_replication_factor=true --catalog_manager_evict_excess_replicas=false --rpc_reuseport=true with env {} WARNING: Logging before InitGoogleLogging() is written to STDERR W0507 06:28:37.592818 2677 flags.cc:404] Enabled unsafe flag: --allow_unsafe_replication_factor=true W0507 06:28:37.593075 2677 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0507 06:28:37.593224 2677 flags.cc:404] Enabled unsafe flag: --never_fsync=true W0507 06:28:37.595875 2677 flags.cc:404] Enabled experimental flag: --ipki_ca_key_size=1024 W0507 06:28:37.595937 2677 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W0507 06:28:37.595969 2677 flags.cc:404] Enabled experimental flag: --tsk_num_rsa_bits=512 W0507 06:28:37.596014 2677 flags.cc:404] Enabled experimental flag: --rpc_reuseport=true I0507 06:28:37.601204 2677 master_main.cc:73] Master server non-default flags: --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.NonVoterReplicasDoNotVote.1557210500371863-231/raft_consensus-itest-cluster/master-0/data --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.NonVoterReplicasDoNotVote.1557210500371863-231/raft_consensus-itest-cluster/master-0/wal --allow_unsafe_replication_factor=true --catalog_manager_evict_excess_replicas=false --ipki_ca_key_size=1024 --ipki_server_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.0.57.254:46409 --rpc_reuseport=true --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.NonVoterReplicasDoNotVote.1557210500371863-231/raft_consensus-itest-cluster/master-0/data/info.pb --webserver_interface=127.0.57.254 --webserver_port=0 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.NonVoterReplicasDoNotVote.1557210500371863-231/raft_consensus-itest-cluster/master-0/logs --logbuflevel=-1 --logtostderr=true Master server version: kudu 1.10.0-cdh6.x-SNAPSHOT revision 7d0ee26697926e87af9ef395008799577f6ab43f build type FASTDEBUG built by jenkins at 06 May 2019 22:44:21 PST on quasar-knmpfx-1.vpc.cloudera.com build id 322017 ASAN enabled I0507 06:28:37.608497 2677 system_ntp.cc:202] NTP initialized. Skew: 500ppm Current error: 33016us I0507 06:28:37.609912 2677 fs_manager.cc:263] Metadata directory not provided I0507 06:28:37.609982 2677 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0507 06:28:37.610327 2677 server_base.cc:433] Could not load existing FS layout: Not found: could not find a healthy instance file I0507 06:28:37.610400 2677 server_base.cc:434] Attempting to create new FS layout instead I0507 06:28:37.613128 2677 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.NonVoterReplicasDoNotVote.1557210500371863-231/raft_consensus-itest-cluster/master-0/data/instance: uuid: "92ffbc77225f48a09f63ca66aa6575b6" format_stamp: "Formatted at 2019-05-07 06:28:37 on dist-test-slave-dist-test-slave-rqp6" I0507 06:28:37.613775 2677 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.NonVoterReplicasDoNotVote.1557210500371863-231/raft_consensus-itest-cluster/master-0/wal/instance: uuid: "92ffbc77225f48a09f63ca66aa6575b6" format_stamp: "Formatted at 2019-05-07 06:28:37 on dist-test-slave-dist-test-slave-rqp6" I0507 06:28:37.621230 2677 fs_manager.cc:503] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s I0507 06:28:37.622428 2677 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0507 06:28:37.623045 2677 file_cache.cc:466] Constructed file cache lbm with capacity 419430 I0507 06:28:37.626454 2677 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s I0507 06:28:37.626541 2677 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.NonVoterReplicasDoNotVote.1557210500371863-231/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.NonVoterReplicasDoNotVote.1557210500371863-231/raft_consensus-itest-cluster/master-0/wal uuid: "92ffbc77225f48a09f63ca66aa6575b6" format_stamp: "Formatted at 2019-05-07 06:28:37 on dist-test-slave-dist-test-slave-rqp6" I0507 06:28:37.626706 2677 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.NonVoterReplicasDoNotVote.1557210500371863-231/raft_consensus-itest-cluster/master-0/wal metadata directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.NonVoterReplicasDoNotVote.1557210500371863-231/raft_consensus-itest-cluster/master-0/wal 1 data directories: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.NonVoterReplicasDoNotVote.1557210500371863-231/raft_consensus-itest-cluster/master-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0507 06:28:37.648893 2677 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0507 06:28:37.664877 2677 rpc_server.cc:205] RPC server started. Bound to: 127.0.57.254:46409 I0507 06:28:37.666070 2677 webserver.cc:308] Webserver started at http://127.0.57.254:41799/ using document root and password file I0507 06:28:37.666887 2677 server_base.cc:608] Dumped server information to /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.NonVoterReplicasDoNotVote.1557210500371863-231/raft_consensus-itest-cluster/master-0/data/info.pb I0507 06:28:37.667299 231 external_mini_cluster.cc:995] Started /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-master as pid 2677 I0507 06:28:37.668000 231 external_mini_cluster.cc:933] Running /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.NonVoterReplicasDoNotVote.1557210500371863-231/raft_consensus-itest-cluster/ts-0/wal --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.NonVoterReplicasDoNotVote.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data --block_manager=log --rpc_bind_addresses=127.0.57.193:0 --local_ip_for_outbound_sockets=127.0.57.193 --webserver_interface=127.0.57.193 --webserver_port=0 --tserver_master_addrs=127.0.57.254:46409 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.NonVoterReplicasDoNotVote.1557210500371863-231/raft_consensus-itest-cluster/ts-0/logs --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.NonVoterReplicasDoNotVote.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_heartbeat_interval_ms=64 --leader_failure_max_missed_heartbeat_periods=1 with env {} I0507 06:28:37.670717 2736 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0507 06:28:37.691215 2736 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 92ffbc77225f48a09f63ca66aa6575b6: Bootstrap starting. I0507 06:28:37.694258 2736 tablet_bootstrap.cc:590] T 00000000000000000000000000000000 P 92ffbc77225f48a09f63ca66aa6575b6: No blocks or log segments found. Creating new log. I0507 06:28:37.695245 2736 log.cc:525] T 00000000000000000000000000000000 P 92ffbc77225f48a09f63ca66aa6575b6: Log is configured to *not* fsync() on all Append() calls I0507 06:28:37.710780 2736 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 92ffbc77225f48a09f63ca66aa6575b6: No bootstrap required, opened a new log I0507 06:28:37.721072 2736 raft_consensus.cc:340] T 00000000000000000000000000000000 P 92ffbc77225f48a09f63ca66aa6575b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92ffbc77225f48a09f63ca66aa6575b6" member_type: VOTER } I0507 06:28:37.721479 2736 raft_consensus.cc:366] T 00000000000000000000000000000000 P 92ffbc77225f48a09f63ca66aa6575b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0507 06:28:37.721583 2736 raft_consensus.cc:705] T 00000000000000000000000000000000 P 92ffbc77225f48a09f63ca66aa6575b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92ffbc77225f48a09f63ca66aa6575b6, State: Initialized, Role: FOLLOWER I0507 06:28:37.722326 2736 consensus_queue.cc:229] T 00000000000000000000000000000000 P 92ffbc77225f48a09f63ca66aa6575b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92ffbc77225f48a09f63ca66aa6575b6" member_type: VOTER } I0507 06:28:37.722550 2736 raft_consensus.cc:380] T 00000000000000000000000000000000 P 92ffbc77225f48a09f63ca66aa6575b6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0507 06:28:37.722638 2736 raft_consensus.cc:472] T 00000000000000000000000000000000 P 92ffbc77225f48a09f63ca66aa6575b6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0507 06:28:37.722754 2736 raft_consensus.cc:2886] T 00000000000000000000000000000000 P 92ffbc77225f48a09f63ca66aa6575b6 [term 0 FOLLOWER]: Advancing to term 1 WARNING: Logging before InitGoogleLogging() is written to STDERR W0507 06:28:37.849066 2737 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0507 06:28:37.849395 2737 flags.cc:404] Enabled unsafe flag: --never_fsync=true W0507 06:28:37.851604 2737 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W0507 06:28:37.851749 2737 flags.cc:404] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.57.193 I0507 06:28:37.856113 2737 tablet_server_main.cc:78] Tablet server non-default flags: --leader_failure_max_missed_heartbeat_periods=1 --raft_heartbeat_interval_ms=64 --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.NonVoterReplicasDoNotVote.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.NonVoterReplicasDoNotVote.1557210500371863-231/raft_consensus-itest-cluster/ts-0/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.0.57.193:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.NonVoterReplicasDoNotVote.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data/info.pb --webserver_interface=127.0.57.193 --webserver_port=0 --tserver_master_addrs=127.0.57.254:46409 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.0.57.193 --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.NonVoterReplicasDoNotVote.1557210500371863-231/raft_consensus-itest-cluster/ts-0/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.10.0-cdh6.x-SNAPSHOT revision 7d0ee26697926e87af9ef395008799577f6ab43f build type FASTDEBUG built by jenkins at 06 May 2019 22:44:21 PST on quasar-knmpfx-1.vpc.cloudera.com build id 322017 ASAN enabled I0507 06:28:37.863940 2737 system_ntp.cc:202] NTP initialized. Skew: 500ppm Current error: 33016us I0507 06:28:37.864817 2737 fs_manager.cc:263] Metadata directory not provided I0507 06:28:37.864902 2737 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0507 06:28:37.865211 2737 server_base.cc:433] Could not load existing FS layout: Not found: could not find a healthy instance file I0507 06:28:37.865293 2737 server_base.cc:434] Attempting to create new FS layout instead I0507 06:28:37.867990 2737 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.NonVoterReplicasDoNotVote.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data/instance: uuid: "0c1b3bd57dab421faa01448383625df1" format_stamp: "Formatted at 2019-05-07 06:28:37 on dist-test-slave-dist-test-slave-rqp6" I0507 06:28:37.868669 2737 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.NonVoterReplicasDoNotVote.1557210500371863-231/raft_consensus-itest-cluster/ts-0/wal/instance: uuid: "0c1b3bd57dab421faa01448383625df1" format_stamp: "Formatted at 2019-05-07 06:28:37 on dist-test-slave-dist-test-slave-rqp6" I0507 06:28:37.875344 2737 fs_manager.cc:503] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s I0507 06:28:37.876425 2737 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0507 06:28:37.877040 2737 file_cache.cc:466] Constructed file cache lbm with capacity 419430 I0507 06:28:37.880152 2737 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s I0507 06:28:37.880220 2737 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.NonVoterReplicasDoNotVote.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.NonVoterReplicasDoNotVote.1557210500371863-231/raft_consensus-itest-cluster/ts-0/wal uuid: "0c1b3bd57dab421faa01448383625df1" format_stamp: "Formatted at 2019-05-07 06:28:37 on dist-test-slave-dist-test-slave-rqp6" I0507 06:28:37.880426 2737 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.NonVoterReplicasDoNotVote.1557210500371863-231/raft_consensus-itest-cluster/ts-0/wal metadata directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.NonVoterReplicasDoNotVote.1557210500371863-231/raft_consensus-itest-cluster/ts-0/wal 1 data directories: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.NonVoterReplicasDoNotVote.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0507 06:28:37.890079 2736 raft_consensus.cc:494] T 00000000000000000000000000000000 P 92ffbc77225f48a09f63ca66aa6575b6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92ffbc77225f48a09f63ca66aa6575b6" member_type: VOTER } I0507 06:28:37.891306 2736 leader_election.cc:310] T 00000000000000000000000000000000 P 92ffbc77225f48a09f63ca66aa6575b6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 92ffbc77225f48a09f63ca66aa6575b6; no voters: I0507 06:28:37.892761 2736 leader_election.cc:296] T 00000000000000000000000000000000 P 92ffbc77225f48a09f63ca66aa6575b6 [CANDIDATE]: Term 1 election: Requested vote from peers I0507 06:28:37.893029 2749 raft_consensus.cc:2640] T 00000000000000000000000000000000 P 92ffbc77225f48a09f63ca66aa6575b6 [term 1 FOLLOWER]: Leader election won for term 1 I0507 06:28:37.894564 2749 raft_consensus.cc:667] T 00000000000000000000000000000000 P 92ffbc77225f48a09f63ca66aa6575b6 [term 1 LEADER]: Becoming Leader. State: Replica: 92ffbc77225f48a09f63ca66aa6575b6, State: Running, Role: LEADER I0507 06:28:37.895028 2749 consensus_queue.cc:206] T 00000000000000000000000000000000 P 92ffbc77225f48a09f63ca66aa6575b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92ffbc77225f48a09f63ca66aa6575b6" member_type: VOTER } I0507 06:28:37.899942 2751 sys_catalog.cc:337] T 00000000000000000000000000000000 P 92ffbc77225f48a09f63ca66aa6575b6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 92ffbc77225f48a09f63ca66aa6575b6. Latest consensus state: current_term: 1 leader_uuid: "92ffbc77225f48a09f63ca66aa6575b6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92ffbc77225f48a09f63ca66aa6575b6" member_type: VOTER } } I0507 06:28:37.900319 2751 sys_catalog.cc:340] T 00000000000000000000000000000000 P 92ffbc77225f48a09f63ca66aa6575b6 [sys.catalog]: This master's current role is: LEADER I0507 06:28:37.901048 2736 sys_catalog.cc:337] T 00000000000000000000000000000000 P 92ffbc77225f48a09f63ca66aa6575b6 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 1 leader_uuid: "92ffbc77225f48a09f63ca66aa6575b6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92ffbc77225f48a09f63ca66aa6575b6" member_type: VOTER } } I0507 06:28:37.901316 2736 sys_catalog.cc:340] T 00000000000000000000000000000000 P 92ffbc77225f48a09f63ca66aa6575b6 [sys.catalog]: This master's current role is: LEADER I0507 06:28:37.900517 2750 sys_catalog.cc:337] T 00000000000000000000000000000000 P 92ffbc77225f48a09f63ca66aa6575b6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "92ffbc77225f48a09f63ca66aa6575b6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92ffbc77225f48a09f63ca66aa6575b6" member_type: VOTER } } I0507 06:28:37.901582 2750 sys_catalog.cc:340] T 00000000000000000000000000000000 P 92ffbc77225f48a09f63ca66aa6575b6 [sys.catalog]: This master's current role is: LEADER I0507 06:28:37.902992 2736 sys_catalog.cc:424] T 00000000000000000000000000000000 P 92ffbc77225f48a09f63ca66aa6575b6 [sys.catalog]: configured and running, proceeding with master startup. I0507 06:28:37.903288 2755 catalog_manager.cc:1055] Loading table and tablet metadata into memory... I0507 06:28:37.903666 2756 catalog_manager.cc:538] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting I0507 06:28:37.908111 2755 catalog_manager.cc:1066] Initializing Kudu internal certificate authority... I0507 06:28:37.921031 2737 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0507 06:28:37.925741 2737 ts_tablet_manager.cc:356] Loaded tablet metadata (0 total tablets, 0 live tablets) I0507 06:28:37.925803 2737 ts_tablet_manager.cc:376] Registered 0 tablets I0507 06:28:37.968006 2755 catalog_manager.cc:950] Generated new certificate authority record I0507 06:28:37.970391 2755 catalog_manager.cc:1075] Loading token signing keys... I0507 06:28:37.989641 2755 catalog_manager.cc:4292] T 00000000000000000000000000000000 P 92ffbc77225f48a09f63ca66aa6575b6: Generated new TSK 0 I0507 06:28:37.995697 2737 rpc_server.cc:205] RPC server started. Bound to: 127.0.57.193:35893 I0507 06:28:37.996985 2737 webserver.cc:308] Webserver started at http://127.0.57.193:34099/ using document root and password file I0507 06:28:37.997714 2737 server_base.cc:608] Dumped server information to /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.NonVoterReplicasDoNotVote.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data/info.pb I0507 06:28:38.005944 231 external_mini_cluster.cc:995] Started /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver as pid 2737 I0507 06:28:38.006779 231 external_mini_cluster.cc:933] Running /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.NonVoterReplicasDoNotVote.1557210500371863-231/raft_consensus-itest-cluster/ts-1/wal --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.NonVoterReplicasDoNotVote.1557210500371863-231/raft_consensus-itest-cluster/ts-1/data --block_manager=log --rpc_bind_addresses=127.0.57.194:0 --local_ip_for_outbound_sockets=127.0.57.194 --webserver_interface=127.0.57.194 --webserver_port=0 --tserver_master_addrs=127.0.57.254:46409 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.NonVoterReplicasDoNotVote.1557210500371863-231/raft_consensus-itest-cluster/ts-1/logs --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.NonVoterReplicasDoNotVote.1557210500371863-231/raft_consensus-itest-cluster/ts-1/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_heartbeat_interval_ms=64 --leader_failure_max_missed_heartbeat_periods=1 with env {} I0507 06:28:38.025558 2867 heartbeater.cc:345] Connected to a master server at 127.0.57.254:46409 I0507 06:28:38.025849 2867 heartbeater.cc:426] Registering TS with master... I0507 06:28:38.026710 2867 heartbeater.cc:475] Master 127.0.57.254:46409 requested a full tablet report, sending... I0507 06:28:38.028806 2703 ts_manager.cc:159] Registered new tserver with Master: 0c1b3bd57dab421faa01448383625df1 (127.0.57.193:35893) I0507 06:28:38.031286 2703 master_service.cc:273] Signed X509 certificate for tserver {username='slave'} at 127.0.57.193:50335 WARNING: Logging before InitGoogleLogging() is written to STDERR W0507 06:28:38.175850 2871 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0507 06:28:38.176069 2871 flags.cc:404] Enabled unsafe flag: --never_fsync=true W0507 06:28:38.178359 2871 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W0507 06:28:38.178469 2871 flags.cc:404] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.57.194 I0507 06:28:38.192940 2871 tablet_server_main.cc:78] Tablet server non-default flags: --leader_failure_max_missed_heartbeat_periods=1 --raft_heartbeat_interval_ms=64 --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.NonVoterReplicasDoNotVote.1557210500371863-231/raft_consensus-itest-cluster/ts-1/data --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.NonVoterReplicasDoNotVote.1557210500371863-231/raft_consensus-itest-cluster/ts-1/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.0.57.194:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.NonVoterReplicasDoNotVote.1557210500371863-231/raft_consensus-itest-cluster/ts-1/data/info.pb --webserver_interface=127.0.57.194 --webserver_port=0 --tserver_master_addrs=127.0.57.254:46409 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.0.57.194 --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.NonVoterReplicasDoNotVote.1557210500371863-231/raft_consensus-itest-cluster/ts-1/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.10.0-cdh6.x-SNAPSHOT revision 7d0ee26697926e87af9ef395008799577f6ab43f build type FASTDEBUG built by jenkins at 06 May 2019 22:44:21 PST on quasar-knmpfx-1.vpc.cloudera.com build id 322017 ASAN enabled I0507 06:28:38.201479 2871 system_ntp.cc:202] NTP initialized. Skew: 500ppm Current error: 33516us I0507 06:28:38.202229 2871 fs_manager.cc:263] Metadata directory not provided I0507 06:28:38.202307 2871 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0507 06:28:38.202575 2871 server_base.cc:433] Could not load existing FS layout: Not found: could not find a healthy instance file I0507 06:28:38.202652 2871 server_base.cc:434] Attempting to create new FS layout instead I0507 06:28:38.205267 2871 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.NonVoterReplicasDoNotVote.1557210500371863-231/raft_consensus-itest-cluster/ts-1/data/instance: uuid: "1f3225ac14a24ce4be60bcf2c130d634" format_stamp: "Formatted at 2019-05-07 06:28:38 on dist-test-slave-dist-test-slave-rqp6" I0507 06:28:38.205868 2871 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.NonVoterReplicasDoNotVote.1557210500371863-231/raft_consensus-itest-cluster/ts-1/wal/instance: uuid: "1f3225ac14a24ce4be60bcf2c130d634" format_stamp: "Formatted at 2019-05-07 06:28:38 on dist-test-slave-dist-test-slave-rqp6" I0507 06:28:38.223572 2871 fs_manager.cc:503] Time spent creating directory manager: real 0.017s user 0.005s sys 0.012s I0507 06:28:38.224882 2871 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0507 06:28:38.225605 2871 file_cache.cc:466] Constructed file cache lbm with capacity 419430 I0507 06:28:38.229136 2871 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.000s sys 0.000s I0507 06:28:38.229208 2871 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.NonVoterReplicasDoNotVote.1557210500371863-231/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.NonVoterReplicasDoNotVote.1557210500371863-231/raft_consensus-itest-cluster/ts-1/wal uuid: "1f3225ac14a24ce4be60bcf2c130d634" format_stamp: "Formatted at 2019-05-07 06:28:38 on dist-test-slave-dist-test-slave-rqp6" I0507 06:28:38.229444 2871 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.NonVoterReplicasDoNotVote.1557210500371863-231/raft_consensus-itest-cluster/ts-1/wal metadata directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.NonVoterReplicasDoNotVote.1557210500371863-231/raft_consensus-itest-cluster/ts-1/wal 1 data directories: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.NonVoterReplicasDoNotVote.1557210500371863-231/raft_consensus-itest-cluster/ts-1/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0507 06:28:38.270218 2871 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0507 06:28:38.272229 2871 ts_tablet_manager.cc:356] Loaded tablet metadata (0 total tablets, 0 live tablets) I0507 06:28:38.272292 2871 ts_tablet_manager.cc:376] Registered 0 tablets I0507 06:28:38.300554 2871 rpc_server.cc:205] RPC server started. Bound to: 127.0.57.194:41551 I0507 06:28:38.301815 2871 webserver.cc:308] Webserver started at http://127.0.57.194:36339/ using document root and password file I0507 06:28:38.302599 2871 server_base.cc:608] Dumped server information to /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.NonVoterReplicasDoNotVote.1557210500371863-231/raft_consensus-itest-cluster/ts-1/data/info.pb I0507 06:28:38.307268 231 external_mini_cluster.cc:995] Started /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver as pid 2871 I0507 06:28:38.307869 231 external_mini_cluster.cc:933] Running /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.NonVoterReplicasDoNotVote.1557210500371863-231/raft_consensus-itest-cluster/ts-2/wal --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.NonVoterReplicasDoNotVote.1557210500371863-231/raft_consensus-itest-cluster/ts-2/data --block_manager=log --rpc_bind_addresses=127.0.57.195:0 --local_ip_for_outbound_sockets=127.0.57.195 --webserver_interface=127.0.57.195 --webserver_port=0 --tserver_master_addrs=127.0.57.254:46409 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.NonVoterReplicasDoNotVote.1557210500371863-231/raft_consensus-itest-cluster/ts-2/logs --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.NonVoterReplicasDoNotVote.1557210500371863-231/raft_consensus-itest-cluster/ts-2/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_heartbeat_interval_ms=64 --leader_failure_max_missed_heartbeat_periods=1 with env {} I0507 06:28:38.320854 2990 heartbeater.cc:345] Connected to a master server at 127.0.57.254:46409 I0507 06:28:38.321070 2990 heartbeater.cc:426] Registering TS with master... I0507 06:28:38.321650 2990 heartbeater.cc:475] Master 127.0.57.254:46409 requested a full tablet report, sending... I0507 06:28:38.322926 2703 ts_manager.cc:159] Registered new tserver with Master: 1f3225ac14a24ce4be60bcf2c130d634 (127.0.57.194:41551) I0507 06:28:38.323865 2703 master_service.cc:273] Signed X509 certificate for tserver {username='slave'} at 127.0.57.194:33529 WARNING: Logging before InitGoogleLogging() is written to STDERR W0507 06:28:38.475626 2993 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0507 06:28:38.475867 2993 flags.cc:404] Enabled unsafe flag: --never_fsync=true W0507 06:28:38.478142 2993 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W0507 06:28:38.478435 2993 flags.cc:404] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.57.195 I0507 06:28:38.482668 2993 tablet_server_main.cc:78] Tablet server non-default flags: --leader_failure_max_missed_heartbeat_periods=1 --raft_heartbeat_interval_ms=64 --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.NonVoterReplicasDoNotVote.1557210500371863-231/raft_consensus-itest-cluster/ts-2/data --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.NonVoterReplicasDoNotVote.1557210500371863-231/raft_consensus-itest-cluster/ts-2/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.0.57.195:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.NonVoterReplicasDoNotVote.1557210500371863-231/raft_consensus-itest-cluster/ts-2/data/info.pb --webserver_interface=127.0.57.195 --webserver_port=0 --tserver_master_addrs=127.0.57.254:46409 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.0.57.195 --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.NonVoterReplicasDoNotVote.1557210500371863-231/raft_consensus-itest-cluster/ts-2/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.10.0-cdh6.x-SNAPSHOT revision 7d0ee26697926e87af9ef395008799577f6ab43f build type FASTDEBUG built by jenkins at 06 May 2019 22:44:21 PST on quasar-knmpfx-1.vpc.cloudera.com build id 322017 ASAN enabled I0507 06:28:38.490748 2993 system_ntp.cc:202] NTP initialized. Skew: 500ppm Current error: 33516us I0507 06:28:38.491457 2993 fs_manager.cc:263] Metadata directory not provided I0507 06:28:38.491540 2993 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0507 06:28:38.491803 2993 server_base.cc:433] Could not load existing FS layout: Not found: could not find a healthy instance file I0507 06:28:38.491879 2993 server_base.cc:434] Attempting to create new FS layout instead I0507 06:28:38.494395 2993 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.NonVoterReplicasDoNotVote.1557210500371863-231/raft_consensus-itest-cluster/ts-2/data/instance: uuid: "9b897477ac24457ba3057221142ea9b4" format_stamp: "Formatted at 2019-05-07 06:28:38 on dist-test-slave-dist-test-slave-rqp6" I0507 06:28:38.494980 2993 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.NonVoterReplicasDoNotVote.1557210500371863-231/raft_consensus-itest-cluster/ts-2/wal/instance: uuid: "9b897477ac24457ba3057221142ea9b4" format_stamp: "Formatted at 2019-05-07 06:28:38 on dist-test-slave-dist-test-slave-rqp6" I0507 06:28:38.501590 2993 fs_manager.cc:503] Time spent creating directory manager: real 0.006s user 0.002s sys 0.006s I0507 06:28:38.502727 2993 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0507 06:28:38.503355 2993 file_cache.cc:466] Constructed file cache lbm with capacity 419430 I0507 06:28:38.506852 2993 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.000s sys 0.000s I0507 06:28:38.506920 2993 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.NonVoterReplicasDoNotVote.1557210500371863-231/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.NonVoterReplicasDoNotVote.1557210500371863-231/raft_consensus-itest-cluster/ts-2/wal uuid: "9b897477ac24457ba3057221142ea9b4" format_stamp: "Formatted at 2019-05-07 06:28:38 on dist-test-slave-dist-test-slave-rqp6" I0507 06:28:38.507160 2993 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.NonVoterReplicasDoNotVote.1557210500371863-231/raft_consensus-itest-cluster/ts-2/wal metadata directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.NonVoterReplicasDoNotVote.1557210500371863-231/raft_consensus-itest-cluster/ts-2/wal 1 data directories: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.NonVoterReplicasDoNotVote.1557210500371863-231/raft_consensus-itest-cluster/ts-2/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0507 06:28:38.556937 2993 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0507 06:28:38.558684 2993 ts_tablet_manager.cc:356] Loaded tablet metadata (0 total tablets, 0 live tablets) I0507 06:28:38.558745 2993 ts_tablet_manager.cc:376] Registered 0 tablets I0507 06:28:38.586354 2993 rpc_server.cc:205] RPC server started. Bound to: 127.0.57.195:38257 I0507 06:28:38.587713 2993 webserver.cc:308] Webserver started at http://127.0.57.195:33383/ using document root and password file I0507 06:28:38.588449 2993 server_base.cc:608] Dumped server information to /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.NonVoterReplicasDoNotVote.1557210500371863-231/raft_consensus-itest-cluster/ts-2/data/info.pb I0507 06:28:38.597559 231 external_mini_cluster.cc:995] Started /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver as pid 2993 I0507 06:28:38.601646 3112 heartbeater.cc:345] Connected to a master server at 127.0.57.254:46409 I0507 06:28:38.601850 3112 heartbeater.cc:426] Registering TS with master... I0507 06:28:38.602401 3112 heartbeater.cc:475] Master 127.0.57.254:46409 requested a full tablet report, sending... I0507 06:28:38.603543 2703 ts_manager.cc:159] Registered new tserver with Master: 9b897477ac24457ba3057221142ea9b4 (127.0.57.195:38257) I0507 06:28:38.604501 2703 master_service.cc:273] Signed X509 certificate for tserver {username='slave'} at 127.0.57.195:45563 I0507 06:28:38.609771 231 external_mini_cluster.cc:623] 3 TS(s) registered with all masters I0507 06:28:38.622057 2703 catalog_manager.cc:1415] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43194: name: "TestTable" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 2 split_rows_range_bounds { } partition_schema { range_schema { columns { name: "key" } } } I0507 06:28:38.651937 3049 tablet_service.cc:1016] Processing CreateTablet for tablet 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d (table=TestTable [id=4d1081c38da64f2e899ec83913895631]), partition=RANGE (key) PARTITION UNBOUNDED I0507 06:28:38.653038 3049 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d. 1 dirs total, 0 dirs full, 0 dirs failed I0507 06:28:38.652920 2803 tablet_service.cc:1016] Processing CreateTablet for tablet 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d (table=TestTable [id=4d1081c38da64f2e899ec83913895631]), partition=RANGE (key) PARTITION UNBOUNDED I0507 06:28:38.653872 2803 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d. 1 dirs total, 0 dirs full, 0 dirs failed I0507 06:28:38.665287 3129 tablet_bootstrap.cc:439] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 9b897477ac24457ba3057221142ea9b4: Bootstrap starting. I0507 06:28:38.667843 3129 tablet_bootstrap.cc:590] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 9b897477ac24457ba3057221142ea9b4: No blocks or log segments found. Creating new log. I0507 06:28:38.670637 3129 log.cc:525] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 9b897477ac24457ba3057221142ea9b4: Log is configured to *not* fsync() on all Append() calls I0507 06:28:38.670753 3130 tablet_bootstrap.cc:439] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 0c1b3bd57dab421faa01448383625df1: Bootstrap starting. I0507 06:28:38.672912 3129 tablet_bootstrap.cc:439] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 9b897477ac24457ba3057221142ea9b4: No bootstrap required, opened a new log I0507 06:28:38.673236 3129 ts_tablet_manager.cc:1053] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 9b897477ac24457ba3057221142ea9b4: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s I0507 06:28:38.673903 3130 tablet_bootstrap.cc:590] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 0c1b3bd57dab421faa01448383625df1: No blocks or log segments found. Creating new log. I0507 06:28:38.675165 3130 log.cc:525] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 0c1b3bd57dab421faa01448383625df1: Log is configured to *not* fsync() on all Append() calls I0507 06:28:38.677357 3130 tablet_bootstrap.cc:439] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 0c1b3bd57dab421faa01448383625df1: No bootstrap required, opened a new log I0507 06:28:38.677645 3130 ts_tablet_manager.cc:1053] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 0c1b3bd57dab421faa01448383625df1: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s I0507 06:28:38.681385 3129 raft_consensus.cc:340] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 9b897477ac24457ba3057221142ea9b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b897477ac24457ba3057221142ea9b4" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 38257 } } peers { permanent_uuid: "0c1b3bd57dab421faa01448383625df1" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 35893 } } I0507 06:28:38.681870 3129 raft_consensus.cc:366] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 9b897477ac24457ba3057221142ea9b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0507 06:28:38.681936 3129 raft_consensus.cc:705] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 9b897477ac24457ba3057221142ea9b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b897477ac24457ba3057221142ea9b4, State: Initialized, Role: FOLLOWER I0507 06:28:38.682667 3129 consensus_queue.cc:229] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 9b897477ac24457ba3057221142ea9b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b897477ac24457ba3057221142ea9b4" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 38257 } } peers { permanent_uuid: "0c1b3bd57dab421faa01448383625df1" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 35893 } } I0507 06:28:38.685165 3112 heartbeater.cc:467] Master 127.0.57.254:46409 was elected leader, sending a full tablet report... I0507 06:28:38.685655 3129 ts_tablet_manager.cc:1081] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 9b897477ac24457ba3057221142ea9b4: Time spent starting tablet: real 0.012s user 0.009s sys 0.003s I0507 06:28:38.686437 3130 raft_consensus.cc:340] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 0c1b3bd57dab421faa01448383625df1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b897477ac24457ba3057221142ea9b4" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 38257 } } peers { permanent_uuid: "0c1b3bd57dab421faa01448383625df1" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 35893 } } I0507 06:28:38.686805 3130 raft_consensus.cc:366] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 0c1b3bd57dab421faa01448383625df1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0507 06:28:38.686877 3130 raft_consensus.cc:705] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 0c1b3bd57dab421faa01448383625df1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c1b3bd57dab421faa01448383625df1, State: Initialized, Role: FOLLOWER I0507 06:28:38.687439 3130 consensus_queue.cc:229] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 0c1b3bd57dab421faa01448383625df1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b897477ac24457ba3057221142ea9b4" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 38257 } } peers { permanent_uuid: "0c1b3bd57dab421faa01448383625df1" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 35893 } } I0507 06:28:38.688403 2867 heartbeater.cc:467] Master 127.0.57.254:46409 was elected leader, sending a full tablet report... I0507 06:28:38.688884 3130 ts_tablet_manager.cc:1081] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 0c1b3bd57dab421faa01448383625df1: Time spent starting tablet: real 0.011s user 0.011s sys 0.000s I0507 06:28:38.744376 3134 raft_consensus.cc:472] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 0c1b3bd57dab421faa01448383625df1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0507 06:28:38.744616 3134 raft_consensus.cc:494] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 0c1b3bd57dab421faa01448383625df1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b897477ac24457ba3057221142ea9b4" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 38257 } } peers { permanent_uuid: "0c1b3bd57dab421faa01448383625df1" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 35893 } } I0507 06:28:38.745734 3133 raft_consensus.cc:472] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 9b897477ac24457ba3057221142ea9b4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0507 06:28:38.745846 3134 leader_election.cc:296] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 0c1b3bd57dab421faa01448383625df1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9b897477ac24457ba3057221142ea9b4 (127.0.57.195:38257) I0507 06:28:38.745918 3133 raft_consensus.cc:494] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 9b897477ac24457ba3057221142ea9b4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b897477ac24457ba3057221142ea9b4" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 38257 } } peers { permanent_uuid: "0c1b3bd57dab421faa01448383625df1" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 35893 } } I0507 06:28:38.746979 3133 leader_election.cc:296] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 9b897477ac24457ba3057221142ea9b4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0c1b3bd57dab421faa01448383625df1 (127.0.57.193:35893) W0507 06:28:38.750492 2868 tablet.cc:1786] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 0c1b3bd57dab421faa01448383625df1: Can't schedule compaction. Clean time has not been advanced past its initial value. I0507 06:28:38.752882 2824 tablet_service.cc:1286] Received RequestConsensusVote() RPC: tablet_id: "9bbbc2f7a1fa40c2a9c8edb97b9fcb5d" candidate_uuid: "9b897477ac24457ba3057221142ea9b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c1b3bd57dab421faa01448383625df1" is_pre_election: true I0507 06:28:38.753455 2824 raft_consensus.cc:2346] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 0c1b3bd57dab421faa01448383625df1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9b897477ac24457ba3057221142ea9b4 in term 0. I0507 06:28:38.754173 3002 leader_election.cc:310] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 9b897477ac24457ba3057221142ea9b4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 0c1b3bd57dab421faa01448383625df1, 9b897477ac24457ba3057221142ea9b4; no voters: I0507 06:28:38.754628 3133 raft_consensus.cc:2640] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 9b897477ac24457ba3057221142ea9b4 [term 0 FOLLOWER]: Leader pre-election won for term 1 I0507 06:28:38.754741 3133 raft_consensus.cc:472] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 9b897477ac24457ba3057221142ea9b4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0507 06:28:38.754820 3133 raft_consensus.cc:2886] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 9b897477ac24457ba3057221142ea9b4 [term 0 FOLLOWER]: Advancing to term 1 I0507 06:28:38.757511 3069 tablet_service.cc:1286] Received RequestConsensusVote() RPC: tablet_id: "9bbbc2f7a1fa40c2a9c8edb97b9fcb5d" candidate_uuid: "0c1b3bd57dab421faa01448383625df1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b897477ac24457ba3057221142ea9b4" is_pre_election: true I0507 06:28:38.824172 3134 raft_consensus.cc:472] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 0c1b3bd57dab421faa01448383625df1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0507 06:28:38.824368 3134 raft_consensus.cc:494] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 0c1b3bd57dab421faa01448383625df1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b897477ac24457ba3057221142ea9b4" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 38257 } } peers { permanent_uuid: "0c1b3bd57dab421faa01448383625df1" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 35893 } } I0507 06:28:38.824920 3134 leader_election.cc:296] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 0c1b3bd57dab421faa01448383625df1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9b897477ac24457ba3057221142ea9b4 (127.0.57.195:38257) I0507 06:28:38.825738 3068 tablet_service.cc:1286] Received RequestConsensusVote() RPC: tablet_id: "9bbbc2f7a1fa40c2a9c8edb97b9fcb5d" candidate_uuid: "0c1b3bd57dab421faa01448383625df1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b897477ac24457ba3057221142ea9b4" is_pre_election: true W0507 06:28:38.836685 2748 leader_election.cc:341] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 0c1b3bd57dab421faa01448383625df1 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 9b897477ac24457ba3057221142ea9b4 (127.0.57.195:38257): Timed out: RequestConsensusVote RPC to 127.0.57.195:38257 timed out after 0.090s (SENT) I0507 06:28:38.836802 2748 leader_election.cc:310] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 0c1b3bd57dab421faa01448383625df1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 0c1b3bd57dab421faa01448383625df1; no voters: 9b897477ac24457ba3057221142ea9b4 I0507 06:28:38.837389 3134 raft_consensus.cc:2588] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 0c1b3bd57dab421faa01448383625df1 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority W0507 06:28:38.839676 3113 tablet.cc:1786] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 9b897477ac24457ba3057221142ea9b4: Can't schedule compaction. Clean time has not been advanced past its initial value. W0507 06:28:38.906183 2748 leader_election.cc:341] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 0c1b3bd57dab421faa01448383625df1 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 9b897477ac24457ba3057221142ea9b4 (127.0.57.195:38257): Timed out: RequestConsensusVote RPC to 127.0.57.195:38257 timed out after 0.080s (SENT) I0507 06:28:38.906297 2748 leader_election.cc:310] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 0c1b3bd57dab421faa01448383625df1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 0c1b3bd57dab421faa01448383625df1; no voters: 9b897477ac24457ba3057221142ea9b4 I0507 06:28:38.906669 3134 raft_consensus.cc:2588] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 0c1b3bd57dab421faa01448383625df1 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority I0507 06:28:39.015375 3134 raft_consensus.cc:472] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 0c1b3bd57dab421faa01448383625df1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0507 06:28:39.015519 3134 raft_consensus.cc:494] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 0c1b3bd57dab421faa01448383625df1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b897477ac24457ba3057221142ea9b4" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 38257 } } peers { permanent_uuid: "0c1b3bd57dab421faa01448383625df1" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 35893 } } I0507 06:28:39.016219 3134 leader_election.cc:296] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 0c1b3bd57dab421faa01448383625df1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9b897477ac24457ba3057221142ea9b4 (127.0.57.195:38257) I0507 06:28:39.017133 3067 tablet_service.cc:1286] Received RequestConsensusVote() RPC: tablet_id: "9bbbc2f7a1fa40c2a9c8edb97b9fcb5d" candidate_uuid: "0c1b3bd57dab421faa01448383625df1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b897477ac24457ba3057221142ea9b4" is_pre_election: true I0507 06:28:39.146719 3134 raft_consensus.cc:472] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 0c1b3bd57dab421faa01448383625df1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0507 06:28:39.146914 3134 raft_consensus.cc:494] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 0c1b3bd57dab421faa01448383625df1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b897477ac24457ba3057221142ea9b4" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 38257 } } peers { permanent_uuid: "0c1b3bd57dab421faa01448383625df1" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 35893 } } W0507 06:28:39.147182 2748 leader_election.cc:341] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 0c1b3bd57dab421faa01448383625df1 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 9b897477ac24457ba3057221142ea9b4 (127.0.57.195:38257): Timed out: RequestConsensusVote RPC to 127.0.57.195:38257 timed out after 0.130s (SENT) I0507 06:28:39.147262 2748 leader_election.cc:310] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 0c1b3bd57dab421faa01448383625df1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 0c1b3bd57dab421faa01448383625df1; no voters: 9b897477ac24457ba3057221142ea9b4 I0507 06:28:39.147712 3134 leader_election.cc:296] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 0c1b3bd57dab421faa01448383625df1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9b897477ac24457ba3057221142ea9b4 (127.0.57.195:38257) I0507 06:28:39.147876 3134 raft_consensus.cc:2588] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 0c1b3bd57dab421faa01448383625df1 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority I0507 06:28:39.148739 3066 tablet_service.cc:1286] Received RequestConsensusVote() RPC: tablet_id: "9bbbc2f7a1fa40c2a9c8edb97b9fcb5d" candidate_uuid: "0c1b3bd57dab421faa01448383625df1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b897477ac24457ba3057221142ea9b4" is_pre_election: true I0507 06:28:39.251188 3134 raft_consensus.cc:472] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 0c1b3bd57dab421faa01448383625df1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0507 06:28:39.251369 3134 raft_consensus.cc:494] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 0c1b3bd57dab421faa01448383625df1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b897477ac24457ba3057221142ea9b4" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 38257 } } peers { permanent_uuid: "0c1b3bd57dab421faa01448383625df1" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 35893 } } I0507 06:28:39.251947 3134 leader_election.cc:296] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 0c1b3bd57dab421faa01448383625df1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9b897477ac24457ba3057221142ea9b4 (127.0.57.195:38257) I0507 06:28:39.252876 3065 tablet_service.cc:1286] Received RequestConsensusVote() RPC: tablet_id: "9bbbc2f7a1fa40c2a9c8edb97b9fcb5d" candidate_uuid: "0c1b3bd57dab421faa01448383625df1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b897477ac24457ba3057221142ea9b4" is_pre_election: true I0507 06:28:39.326171 2990 heartbeater.cc:467] Master 127.0.57.254:46409 was elected leader, sending a full tablet report... W0507 06:28:39.333940 2748 leader_election.cc:341] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 0c1b3bd57dab421faa01448383625df1 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 9b897477ac24457ba3057221142ea9b4 (127.0.57.195:38257): Timed out: RequestConsensusVote RPC to 127.0.57.195:38257 timed out after 0.185s (SENT) I0507 06:28:39.334105 2748 leader_election.cc:310] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 0c1b3bd57dab421faa01448383625df1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 0c1b3bd57dab421faa01448383625df1; no voters: 9b897477ac24457ba3057221142ea9b4 I0507 06:28:39.334497 3134 raft_consensus.cc:2588] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 0c1b3bd57dab421faa01448383625df1 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority W0507 06:28:39.354861 2748 leader_election.cc:341] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 0c1b3bd57dab421faa01448383625df1 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 9b897477ac24457ba3057221142ea9b4 (127.0.57.195:38257): Timed out: RequestConsensusVote RPC to 127.0.57.195:38257 timed out after 0.102s (SENT) I0507 06:28:39.354970 2748 leader_election.cc:310] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 0c1b3bd57dab421faa01448383625df1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 0c1b3bd57dab421faa01448383625df1; no voters: 9b897477ac24457ba3057221142ea9b4 I0507 06:28:39.355410 3134 raft_consensus.cc:2588] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 0c1b3bd57dab421faa01448383625df1 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority W0507 06:28:39.398620 3133 consensus_meta.cc:220] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 9b897477ac24457ba3057221142ea9b4: Time spent flushing consensus metadata: real 0.644s user 0.002s sys 0.000s I0507 06:28:39.398753 3133 raft_consensus.cc:494] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 9b897477ac24457ba3057221142ea9b4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b897477ac24457ba3057221142ea9b4" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 38257 } } peers { permanent_uuid: "0c1b3bd57dab421faa01448383625df1" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 35893 } } I0507 06:28:39.399271 3068 raft_consensus.cc:2321] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 9b897477ac24457ba3057221142ea9b4 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0c1b3bd57dab421faa01448383625df1 for term 1 because replica is already servicing an update from a current leader or another vote. I0507 06:28:39.399521 3133 leader_election.cc:296] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 9b897477ac24457ba3057221142ea9b4 [CANDIDATE]: Term 1 election: Requested vote from peers 0c1b3bd57dab421faa01448383625df1 (127.0.57.193:35893) I0507 06:28:39.399824 3067 raft_consensus.cc:2321] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 9b897477ac24457ba3057221142ea9b4 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0c1b3bd57dab421faa01448383625df1 for term 1 because replica is already servicing an update from a current leader or another vote. I0507 06:28:39.400043 3069 raft_consensus.cc:2276] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 9b897477ac24457ba3057221142ea9b4 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0c1b3bd57dab421faa01448383625df1 in current term 1: Already voted for candidate 9b897477ac24457ba3057221142ea9b4 in this term. W0507 06:28:39.400295 3068 rpcz_store.cc:253] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.57.193:37955 (request call id 2) took 574 ms (574 ms). Client timeout 80 ms (80 ms) I0507 06:28:39.400504 2824 tablet_service.cc:1286] Received RequestConsensusVote() RPC: tablet_id: "9bbbc2f7a1fa40c2a9c8edb97b9fcb5d" candidate_uuid: "9b897477ac24457ba3057221142ea9b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c1b3bd57dab421faa01448383625df1" I0507 06:28:39.400745 2824 raft_consensus.cc:2886] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 0c1b3bd57dab421faa01448383625df1 [term 0 FOLLOWER]: Advancing to term 1 I0507 06:28:39.400176 3136 raft_consensus.cc:472] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 9b897477ac24457ba3057221142ea9b4 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0507 06:28:39.401095 3136 raft_consensus.cc:494] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 9b897477ac24457ba3057221142ea9b4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b897477ac24457ba3057221142ea9b4" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 38257 } } peers { permanent_uuid: "0c1b3bd57dab421faa01448383625df1" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 35893 } } W0507 06:28:39.401574 3068 rpcz_store.cc:259] Trace: 0507 06:28:38.825435 (+ 0us) service_pool.cc:162] Inserting onto call queue 0507 06:28:38.825536 (+ 101us) service_pool.cc:221] Handling call 0507 06:28:39.399957 (+574421us) spinlock_profiling.cc:243] Waited 573 ms on lock 0x6140000720c4. stack: 00007fd1cc567545 00007fd1d924babd 00007fd1d609d5f4 00007fd1d931790a 00007fd1d0599009 00007fd1cf5835eb 00007fd1cf58205a 00007fd1cf586077 00007fd1cf58b952 00007fd1cf4e0d2b 00007fd1cc58fc6a 00007fd1d292f183 00007fd1c7963ffc 0507 06:28:39.400143 (+ 186us) inbound_call.cc:162] Queueing success response Metrics: {"spinlock_wait_cycles":1318445312} I0507 06:28:39.402060 3065 raft_consensus.cc:2321] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 9b897477ac24457ba3057221142ea9b4 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0c1b3bd57dab421faa01448383625df1 for term 1 because replica is already servicing an update from a current leader or another vote. W0507 06:28:39.402120 3069 rpcz_store.cc:253] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.57.193:37955 (request call id 1) took 643 ms (643 ms). Client timeout 90 ms (90 ms) W0507 06:28:39.402343 3065 rpcz_store.cc:253] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.57.193:37955 (request call id 5) took 149 ms (149 ms). Client timeout 102 ms (102 ms) W0507 06:28:39.402024 3067 rpcz_store.cc:253] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.57.193:37955 (request call id 3) took 384 ms (384 ms). Client timeout 130 ms (130 ms) W0507 06:28:39.402582 3065 rpcz_store.cc:259] Trace: 0507 06:28:39.252601 (+ 0us) service_pool.cc:162] Inserting onto call queue 0507 06:28:39.252689 (+ 88us) service_pool.cc:221] Handling call 0507 06:28:39.402172 (+149483us) spinlock_profiling.cc:243] Waited 148 ms on lock 0x6140000720c4. stack: 00007fd1cc567545 00007fd1d924babd 00007fd1d609d5f4 00007fd1d931790a 00007fd1d0599009 00007fd1cf5835eb 00007fd1cf58205a 00007fd1cf586077 00007fd1cf58b952 00007fd1cf4e0d2b 00007fd1cc58fc6a 00007fd1d292f183 00007fd1c7963ffc 0507 06:28:39.402240 (+ 68us) inbound_call.cc:162] Queueing success response Metrics: {"spinlock_wait_cycles":341394688} W0507 06:28:39.402725 3069 rpcz_store.cc:259] Trace: 0507 06:28:38.757163 (+ 0us) service_pool.cc:162] Inserting onto call queue 0507 06:28:38.757263 (+ 100us) service_pool.cc:221] Handling call 0507 06:28:39.400792 (+643529us) spinlock_profiling.cc:243] Waited 642 ms on lock 0x6140000720c4. stack: 00007fd1cc567545 00007fd1d924babd 00007fd1d609e6e0 00007fd1d931790a 00007fd1d0599009 00007fd1cf5835eb 00007fd1cf58205a 00007fd1cf586077 00007fd1cf58b952 00007fd1cf4e0d2b 00007fd1cc58fc6a 00007fd1d292f183 00007fd1c7963ffc 0507 06:28:39.400953 (+ 161us) inbound_call.cc:162] Queueing success response Metrics: {"spinlock_wait_cycles":1477027584} I0507 06:28:39.402838 3066 raft_consensus.cc:2321] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 9b897477ac24457ba3057221142ea9b4 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0c1b3bd57dab421faa01448383625df1 for term 1 because replica is already servicing an update from a current leader or another vote. I0507 06:28:39.402947 2823 tablet_service.cc:1286] Received RequestConsensusVote() RPC: tablet_id: "9bbbc2f7a1fa40c2a9c8edb97b9fcb5d" candidate_uuid: "9b897477ac24457ba3057221142ea9b4" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c1b3bd57dab421faa01448383625df1" is_pre_election: true I0507 06:28:39.403199 2824 raft_consensus.cc:2346] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 0c1b3bd57dab421faa01448383625df1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9b897477ac24457ba3057221142ea9b4 in term 1. I0507 06:28:39.403363 2823 raft_consensus.cc:2321] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 0c1b3bd57dab421faa01448383625df1 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9b897477ac24457ba3057221142ea9b4 for term 2 because replica is already servicing an update from a current leader or another vote. W0507 06:28:39.403381 3067 rpcz_store.cc:259] Trace: 0507 06:28:39.016703 (+ 0us) service_pool.cc:162] Inserting onto call queue 0507 06:28:39.016838 (+ 135us) service_pool.cc:221] Handling call 0507 06:28:39.400359 (+383521us) spinlock_profiling.cc:243] Waited 382 ms on lock 0x6140000720c4. stack: 00007fd1cc567545 00007fd1d924babd 00007fd1d609d5f4 00007fd1d931790a 00007fd1d0599009 00007fd1cf5835eb 00007fd1cf58205a 00007fd1cf586077 00007fd1cf58b952 00007fd1cf4e0d2b 00007fd1cc58fc6a 00007fd1d292f183 00007fd1c7963ffc 0507 06:28:39.400881 (+ 522us) inbound_call.cc:162] Queueing success response Metrics: {"spinlock_wait_cycles":879639424} I0507 06:28:39.403604 3136 leader_election.cc:296] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 9b897477ac24457ba3057221142ea9b4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0c1b3bd57dab421faa01448383625df1 (127.0.57.193:35893) I0507 06:28:39.403620 3002 leader_election.cc:310] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 9b897477ac24457ba3057221142ea9b4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 0c1b3bd57dab421faa01448383625df1, 9b897477ac24457ba3057221142ea9b4; no voters: W0507 06:28:39.403810 3066 rpcz_store.cc:253] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.57.193:37955 (request call id 4) took 254 ms (254 ms). Client timeout 185 ms (185 ms) W0507 06:28:39.404150 3066 rpcz_store.cc:259] Trace: 0507 06:28:39.148422 (+ 0us) service_pool.cc:162] Inserting onto call queue 0507 06:28:39.148555 (+ 133us) service_pool.cc:221] Handling call 0507 06:28:39.402943 (+254388us) spinlock_profiling.cc:243] Waited 253 ms on lock 0x6140000720c4. stack: 00007fd1cc567545 00007fd1d924babd 00007fd1d609d5f4 00007fd1d931790a 00007fd1d0599009 00007fd1cf5835eb 00007fd1cf58205a 00007fd1cf586077 00007fd1cf58b952 00007fd1cf4e0d2b 00007fd1cc58fc6a 00007fd1d292f183 00007fd1c7963ffc 0507 06:28:39.403033 (+ 90us) inbound_call.cc:162] Queueing success response Metrics: {"spinlock_wait_cycles":582257408} I0507 06:28:39.404189 3136 raft_consensus.cc:2640] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 9b897477ac24457ba3057221142ea9b4 [term 1 FOLLOWER]: Leader election won for term 1 I0507 06:28:39.404336 3136 raft_consensus.cc:667] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 9b897477ac24457ba3057221142ea9b4 [term 1 LEADER]: Becoming Leader. State: Replica: 9b897477ac24457ba3057221142ea9b4, State: Running, Role: LEADER I0507 06:28:39.404372 3002 leader_election.cc:310] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 9b897477ac24457ba3057221142ea9b4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 9b897477ac24457ba3057221142ea9b4; no voters: 0c1b3bd57dab421faa01448383625df1 I0507 06:28:39.404749 3136 consensus_queue.cc:206] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 9b897477ac24457ba3057221142ea9b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b897477ac24457ba3057221142ea9b4" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 38257 } } peers { permanent_uuid: "0c1b3bd57dab421faa01448383625df1" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 35893 } } I0507 06:28:39.406548 3137 raft_consensus.cc:2588] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 9b897477ac24457ba3057221142ea9b4 [term 1 LEADER]: Leader pre-election lost for term 2. Reason: could not achieve majority I0507 06:28:39.408407 2702 catalog_manager.cc:3988] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 9b897477ac24457ba3057221142ea9b4 reported cstate change: term changed from 0 to 1, leader changed from to 9b897477ac24457ba3057221142ea9b4 (127.0.57.195). New cstate: current_term: 1 leader_uuid: "9b897477ac24457ba3057221142ea9b4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b897477ac24457ba3057221142ea9b4" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 38257 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0c1b3bd57dab421faa01448383625df1" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 35893 } health_report { overall_health: UNKNOWN } } } I0507 06:28:39.472200 2823 raft_consensus.cc:1181] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 0c1b3bd57dab421faa01448383625df1 [term 1 FOLLOWER]: Refusing update from remote peer 9b897477ac24457ba3057221142ea9b4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch) I0507 06:28:39.473274 3137 consensus_queue.cc:984] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 9b897477ac24457ba3057221142ea9b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c1b3bd57dab421faa01448383625df1" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 35893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0507 06:28:39.525075 231 external_mini_cluster.cc:623] 3 TS(s) registered with all masters I0507 06:28:39.526825 231 ts_itest-base.cc:232] Waiting for 1 tablets on tserver 0c1b3bd57dab421faa01448383625df1 to finish bootstrapping I0507 06:28:39.534621 231 ts_itest-base.cc:232] Waiting for 0 tablets on tserver 1f3225ac14a24ce4be60bcf2c130d634 to finish bootstrapping I0507 06:28:39.548462 231 ts_itest-base.cc:232] Waiting for 1 tablets on tserver 9b897477ac24457ba3057221142ea9b4 to finish bootstrapping I0507 06:28:39.579423 3066 consensus_queue.cc:206] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 9b897477ac24457ba3057221142ea9b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9b897477ac24457ba3057221142ea9b4" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 38257 } } peers { permanent_uuid: "0c1b3bd57dab421faa01448383625df1" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 35893 } } peers { permanent_uuid: "1f3225ac14a24ce4be60bcf2c130d634" member_type: NON_VOTER last_known_addr { host: "127.0.57.194" port: 41551 } attrs { } } I0507 06:28:39.581609 2823 raft_consensus.cc:1181] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 0c1b3bd57dab421faa01448383625df1 [term 1 FOLLOWER]: Refusing update from remote peer 9b897477ac24457ba3057221142ea9b4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0507 06:28:39.582765 3137 consensus_queue.cc:984] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 9b897477ac24457ba3057221142ea9b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c1b3bd57dab421faa01448383625df1" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 35893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s I0507 06:28:39.585434 3136 raft_consensus.cc:2788] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 9b897477ac24457ba3057221142ea9b4 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1f3225ac14a24ce4be60bcf2c130d634 (127.0.57.194) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9b897477ac24457ba3057221142ea9b4" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 38257 } } peers { permanent_uuid: "0c1b3bd57dab421faa01448383625df1" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 35893 } } peers { permanent_uuid: "1f3225ac14a24ce4be60bcf2c130d634" member_type: NON_VOTER last_known_addr { host: "127.0.57.194" port: 41551 } attrs { } } } I0507 06:28:39.586036 2823 raft_consensus.cc:2788] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 0c1b3bd57dab421faa01448383625df1 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1f3225ac14a24ce4be60bcf2c130d634 (127.0.57.194) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9b897477ac24457ba3057221142ea9b4" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 38257 } } peers { permanent_uuid: "0c1b3bd57dab421faa01448383625df1" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 35893 } } peers { permanent_uuid: "1f3225ac14a24ce4be60bcf2c130d634" member_type: NON_VOTER last_known_addr { host: "127.0.57.194" port: 41551 } attrs { } } } W0507 06:28:39.588301 3001 consensus_peers.cc:457] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 9b897477ac24457ba3057221142ea9b4 -> Peer 1f3225ac14a24ce4be60bcf2c130d634 (127.0.57.194:41551): Couldn't send request to peer 1f3225ac14a24ce4be60bcf2c130d634. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d. This is attempt 1: this message will repeat every 5th retry. I0507 06:28:39.644381 3152 ts_tablet_manager.cc:676] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 1f3225ac14a24ce4be60bcf2c130d634: Initiating tablet copy from peer 9b897477ac24457ba3057221142ea9b4 (127.0.57.195:38257) I0507 06:28:39.644775 3152 tablet_copy_client.cc:240] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 1f3225ac14a24ce4be60bcf2c130d634: tablet copy: Beginning tablet copy session from remote peer at address 127.0.57.195:38257 I0507 06:28:39.651101 3089 tablet_copy_service.cc:135] P 9b897477ac24457ba3057221142ea9b4: Received BeginTabletCopySession request for tablet 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d from peer 1f3225ac14a24ce4be60bcf2c130d634 ({username='slave'} at 127.0.57.194:59197) I0507 06:28:39.651306 3089 tablet_copy_service.cc:156] P 9b897477ac24457ba3057221142ea9b4: Beginning new tablet copy session on tablet 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d from peer 1f3225ac14a24ce4be60bcf2c130d634 at {username='slave'} at 127.0.57.194:59197: session id = 1f3225ac14a24ce4be60bcf2c130d634-9bbbc2f7a1fa40c2a9c8edb97b9fcb5d W0507 06:28:40.306144 2823 consensus_meta.cc:220] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 0c1b3bd57dab421faa01448383625df1: Time spent flushing consensus metadata: real 0.719s user 0.002s sys 0.000s W0507 06:28:40.306294 3136 consensus_meta.cc:220] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 9b897477ac24457ba3057221142ea9b4: Time spent flushing consensus metadata: real 0.720s user 0.001s sys 0.002s I0507 06:28:40.306782 3134 raft_consensus.cc:472] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 0c1b3bd57dab421faa01448383625df1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9b897477ac24457ba3057221142ea9b4) I0507 06:28:40.306980 3134 raft_consensus.cc:494] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 0c1b3bd57dab421faa01448383625df1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9b897477ac24457ba3057221142ea9b4" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 38257 } } peers { permanent_uuid: "0c1b3bd57dab421faa01448383625df1" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 35893 } } peers { permanent_uuid: "1f3225ac14a24ce4be60bcf2c130d634" member_type: NON_VOTER last_known_addr { host: "127.0.57.194" port: 41551 } attrs { } } I0507 06:28:40.307718 3134 leader_election.cc:296] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 0c1b3bd57dab421faa01448383625df1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9b897477ac24457ba3057221142ea9b4 (127.0.57.195:38257) I0507 06:28:40.308744 3066 tablet_service.cc:1286] Received RequestConsensusVote() RPC: tablet_id: "9bbbc2f7a1fa40c2a9c8edb97b9fcb5d" candidate_uuid: "0c1b3bd57dab421faa01448383625df1" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "9b897477ac24457ba3057221142ea9b4" is_pre_election: true I0507 06:28:40.309213 2703 catalog_manager.cc:3988] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 0c1b3bd57dab421faa01448383625df1 reported cstate change: config changed from index -1 to 2, NON_VOTER 1f3225ac14a24ce4be60bcf2c130d634 (127.0.57.194) added. New cstate: current_term: 1 leader_uuid: "9b897477ac24457ba3057221142ea9b4" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9b897477ac24457ba3057221142ea9b4" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 38257 } } peers { permanent_uuid: "0c1b3bd57dab421faa01448383625df1" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 35893 } } peers { permanent_uuid: "1f3225ac14a24ce4be60bcf2c130d634" member_type: NON_VOTER last_known_addr { host: "127.0.57.194" port: 41551 } attrs { } } } I0507 06:28:40.312746 3066 raft_consensus.cc:2307] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 9b897477ac24457ba3057221142ea9b4 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate 0c1b3bd57dab421faa01448383625df1 for term 2 because replica is either leader or believes a valid leader to be alive. I0507 06:28:40.312875 3089 tablet_copy_source_session.cc:202] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 9b897477ac24457ba3057221142ea9b4: Tablet Copy: opened 0 blocks and 1 log segments I0507 06:28:40.313948 2748 leader_election.cc:310] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 0c1b3bd57dab421faa01448383625df1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 0c1b3bd57dab421faa01448383625df1; no voters: 9b897477ac24457ba3057221142ea9b4 I0507 06:28:40.318022 3152 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d. 1 dirs total, 0 dirs full, 0 dirs failed I0507 06:28:40.319770 3154 raft_consensus.cc:2588] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 0c1b3bd57dab421faa01448383625df1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority I0507 06:28:40.329980 3152 tablet_copy_client.cc:575] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 1f3225ac14a24ce4be60bcf2c130d634: tablet copy: Starting download of 0 data blocks... I0507 06:28:40.330317 3152 tablet_copy_client.cc:538] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 1f3225ac14a24ce4be60bcf2c130d634: tablet copy: Starting download of 1 WAL segments... I0507 06:28:40.332370 3152 tablet_copy_client.cc:414] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 1f3225ac14a24ce4be60bcf2c130d634: tablet copy: Tablet Copy complete. Replacing tablet superblock. I0507 06:28:40.427032 3154 raft_consensus.cc:472] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 0c1b3bd57dab421faa01448383625df1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9b897477ac24457ba3057221142ea9b4) I0507 06:28:40.427283 3154 raft_consensus.cc:494] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 0c1b3bd57dab421faa01448383625df1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9b897477ac24457ba3057221142ea9b4" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 38257 } } peers { permanent_uuid: "0c1b3bd57dab421faa01448383625df1" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 35893 } } peers { permanent_uuid: "1f3225ac14a24ce4be60bcf2c130d634" member_type: NON_VOTER last_known_addr { host: "127.0.57.194" port: 41551 } attrs { } } I0507 06:28:40.428143 3154 leader_election.cc:296] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 0c1b3bd57dab421faa01448383625df1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9b897477ac24457ba3057221142ea9b4 (127.0.57.195:38257) I0507 06:28:40.428548 3066 tablet_service.cc:1286] Received RequestConsensusVote() RPC: tablet_id: "9bbbc2f7a1fa40c2a9c8edb97b9fcb5d" candidate_uuid: "0c1b3bd57dab421faa01448383625df1" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "9b897477ac24457ba3057221142ea9b4" is_pre_election: true I0507 06:28:40.428843 3066 raft_consensus.cc:2307] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 9b897477ac24457ba3057221142ea9b4 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate 0c1b3bd57dab421faa01448383625df1 for term 2 because replica is either leader or believes a valid leader to be alive. I0507 06:28:40.429378 2748 leader_election.cc:310] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 0c1b3bd57dab421faa01448383625df1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 0c1b3bd57dab421faa01448383625df1; no voters: 9b897477ac24457ba3057221142ea9b4 I0507 06:28:40.429742 3154 raft_consensus.cc:2588] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 0c1b3bd57dab421faa01448383625df1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority W0507 06:28:40.619026 3001 consensus_peers.cc:457] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 9b897477ac24457ba3057221142ea9b4 -> Peer 1f3225ac14a24ce4be60bcf2c130d634 (127.0.57.194:41551): Couldn't send request to peer 1f3225ac14a24ce4be60bcf2c130d634. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 6: this message will repeat every 5th retry. I0507 06:28:40.819072 3154 raft_consensus.cc:472] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 0c1b3bd57dab421faa01448383625df1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9b897477ac24457ba3057221142ea9b4) I0507 06:28:40.819259 3154 raft_consensus.cc:494] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 0c1b3bd57dab421faa01448383625df1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9b897477ac24457ba3057221142ea9b4" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 38257 } } peers { permanent_uuid: "0c1b3bd57dab421faa01448383625df1" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 35893 } } peers { permanent_uuid: "1f3225ac14a24ce4be60bcf2c130d634" member_type: NON_VOTER last_known_addr { host: "127.0.57.194" port: 41551 } attrs { } } I0507 06:28:40.819805 3154 leader_election.cc:296] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 0c1b3bd57dab421faa01448383625df1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9b897477ac24457ba3057221142ea9b4 (127.0.57.195:38257) I0507 06:28:40.820776 3066 tablet_service.cc:1286] Received RequestConsensusVote() RPC: tablet_id: "9bbbc2f7a1fa40c2a9c8edb97b9fcb5d" candidate_uuid: "0c1b3bd57dab421faa01448383625df1" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "9b897477ac24457ba3057221142ea9b4" is_pre_election: true I0507 06:28:40.821096 3066 raft_consensus.cc:2307] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 9b897477ac24457ba3057221142ea9b4 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate 0c1b3bd57dab421faa01448383625df1 for term 2 because replica is either leader or believes a valid leader to be alive. I0507 06:28:40.821593 2748 leader_election.cc:310] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 0c1b3bd57dab421faa01448383625df1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 0c1b3bd57dab421faa01448383625df1; no voters: 9b897477ac24457ba3057221142ea9b4 I0507 06:28:40.821945 3154 raft_consensus.cc:2588] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 0c1b3bd57dab421faa01448383625df1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority I0507 06:28:40.895831 3154 raft_consensus.cc:472] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 0c1b3bd57dab421faa01448383625df1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9b897477ac24457ba3057221142ea9b4) I0507 06:28:40.896052 3154 raft_consensus.cc:494] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 0c1b3bd57dab421faa01448383625df1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9b897477ac24457ba3057221142ea9b4" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 38257 } } peers { permanent_uuid: "0c1b3bd57dab421faa01448383625df1" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 35893 } } peers { permanent_uuid: "1f3225ac14a24ce4be60bcf2c130d634" member_type: NON_VOTER last_known_addr { host: "127.0.57.194" port: 41551 } attrs { } } I0507 06:28:40.896705 3154 leader_election.cc:296] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 0c1b3bd57dab421faa01448383625df1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9b897477ac24457ba3057221142ea9b4 (127.0.57.195:38257) I0507 06:28:40.897446 3066 tablet_service.cc:1286] Received RequestConsensusVote() RPC: tablet_id: "9bbbc2f7a1fa40c2a9c8edb97b9fcb5d" candidate_uuid: "0c1b3bd57dab421faa01448383625df1" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "9b897477ac24457ba3057221142ea9b4" is_pre_election: true I0507 06:28:40.897691 3066 raft_consensus.cc:2307] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 9b897477ac24457ba3057221142ea9b4 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate 0c1b3bd57dab421faa01448383625df1 for term 2 because replica is either leader or believes a valid leader to be alive. I0507 06:28:40.898231 2748 leader_election.cc:310] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 0c1b3bd57dab421faa01448383625df1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 0c1b3bd57dab421faa01448383625df1; no voters: 9b897477ac24457ba3057221142ea9b4 I0507 06:28:40.898563 3154 raft_consensus.cc:2588] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 0c1b3bd57dab421faa01448383625df1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority W0507 06:28:40.934990 3001 consensus_peers.cc:457] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 9b897477ac24457ba3057221142ea9b4 -> Peer 1f3225ac14a24ce4be60bcf2c130d634 (127.0.57.194:41551): Couldn't send request to peer 1f3225ac14a24ce4be60bcf2c130d634. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 11: this message will repeat every 5th retry. I0507 06:28:41.087496 3154 raft_consensus.cc:472] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 0c1b3bd57dab421faa01448383625df1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9b897477ac24457ba3057221142ea9b4) I0507 06:28:41.087702 3154 raft_consensus.cc:494] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 0c1b3bd57dab421faa01448383625df1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9b897477ac24457ba3057221142ea9b4" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 38257 } } peers { permanent_uuid: "0c1b3bd57dab421faa01448383625df1" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 35893 } } peers { permanent_uuid: "1f3225ac14a24ce4be60bcf2c130d634" member_type: NON_VOTER last_known_addr { host: "127.0.57.194" port: 41551 } attrs { } } I0507 06:28:41.088434 3154 leader_election.cc:296] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 0c1b3bd57dab421faa01448383625df1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9b897477ac24457ba3057221142ea9b4 (127.0.57.195:38257) I0507 06:28:41.089190 3066 tablet_service.cc:1286] Received RequestConsensusVote() RPC: tablet_id: "9bbbc2f7a1fa40c2a9c8edb97b9fcb5d" candidate_uuid: "0c1b3bd57dab421faa01448383625df1" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "9b897477ac24457ba3057221142ea9b4" is_pre_election: true I0507 06:28:41.089421 3066 raft_consensus.cc:2307] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 9b897477ac24457ba3057221142ea9b4 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate 0c1b3bd57dab421faa01448383625df1 for term 2 because replica is either leader or believes a valid leader to be alive. I0507 06:28:41.089931 2748 leader_election.cc:310] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 0c1b3bd57dab421faa01448383625df1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 0c1b3bd57dab421faa01448383625df1; no voters: 9b897477ac24457ba3057221142ea9b4 I0507 06:28:41.090240 3154 raft_consensus.cc:2588] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 0c1b3bd57dab421faa01448383625df1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority I0507 06:28:41.166146 3154 raft_consensus.cc:472] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 0c1b3bd57dab421faa01448383625df1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9b897477ac24457ba3057221142ea9b4) I0507 06:28:41.166345 3154 raft_consensus.cc:494] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 0c1b3bd57dab421faa01448383625df1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9b897477ac24457ba3057221142ea9b4" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 38257 } } peers { permanent_uuid: "0c1b3bd57dab421faa01448383625df1" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 35893 } } peers { permanent_uuid: "1f3225ac14a24ce4be60bcf2c130d634" member_type: NON_VOTER last_known_addr { host: "127.0.57.194" port: 41551 } attrs { } } I0507 06:28:41.166937 3154 leader_election.cc:296] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 0c1b3bd57dab421faa01448383625df1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9b897477ac24457ba3057221142ea9b4 (127.0.57.195:38257) I0507 06:28:41.167697 3066 tablet_service.cc:1286] Received RequestConsensusVote() RPC: tablet_id: "9bbbc2f7a1fa40c2a9c8edb97b9fcb5d" candidate_uuid: "0c1b3bd57dab421faa01448383625df1" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "9b897477ac24457ba3057221142ea9b4" is_pre_election: true I0507 06:28:41.167958 3066 raft_consensus.cc:2307] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 9b897477ac24457ba3057221142ea9b4 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate 0c1b3bd57dab421faa01448383625df1 for term 2 because replica is either leader or believes a valid leader to be alive. I0507 06:28:41.168437 2748 leader_election.cc:310] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 0c1b3bd57dab421faa01448383625df1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 0c1b3bd57dab421faa01448383625df1; no voters: 9b897477ac24457ba3057221142ea9b4 I0507 06:28:41.168808 3154 raft_consensus.cc:2588] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 0c1b3bd57dab421faa01448383625df1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority I0507 06:28:41.203582 3152 tablet_bootstrap.cc:439] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 1f3225ac14a24ce4be60bcf2c130d634: Bootstrap starting. I0507 06:28:41.209600 3152 log.cc:525] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 1f3225ac14a24ce4be60bcf2c130d634: Log is configured to *not* fsync() on all Append() calls I0507 06:28:41.213146 3152 tablet_bootstrap.cc:439] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 1f3225ac14a24ce4be60bcf2c130d634: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I0507 06:28:41.213588 3152 tablet_bootstrap.cc:439] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 1f3225ac14a24ce4be60bcf2c130d634: Bootstrap complete. I0507 06:28:41.213951 3152 ts_tablet_manager.cc:1053] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 1f3225ac14a24ce4be60bcf2c130d634: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.001s I0507 06:28:41.219751 3152 raft_consensus.cc:340] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 1f3225ac14a24ce4be60bcf2c130d634 [term 1 LEARNER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9b897477ac24457ba3057221142ea9b4" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 38257 } } peers { permanent_uuid: "0c1b3bd57dab421faa01448383625df1" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 35893 } } peers { permanent_uuid: "1f3225ac14a24ce4be60bcf2c130d634" member_type: NON_VOTER last_known_addr { host: "127.0.57.194" port: 41551 } attrs { } } I0507 06:28:41.220157 3152 raft_consensus.cc:705] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 1f3225ac14a24ce4be60bcf2c130d634 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1f3225ac14a24ce4be60bcf2c130d634, State: Initialized, Role: LEARNER I0507 06:28:41.220440 3152 consensus_queue.cc:229] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 1f3225ac14a24ce4be60bcf2c130d634 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9b897477ac24457ba3057221142ea9b4" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 38257 } } peers { permanent_uuid: "0c1b3bd57dab421faa01448383625df1" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 35893 } } peers { permanent_uuid: "1f3225ac14a24ce4be60bcf2c130d634" member_type: NON_VOTER last_known_addr { host: "127.0.57.194" port: 41551 } attrs { } } I0507 06:28:41.221822 3152 ts_tablet_manager.cc:1081] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 1f3225ac14a24ce4be60bcf2c130d634: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s I0507 06:28:41.222820 3089 tablet_copy_service.cc:325] P 9b897477ac24457ba3057221142ea9b4: Request end of tablet copy session 1f3225ac14a24ce4be60bcf2c130d634-9bbbc2f7a1fa40c2a9c8edb97b9fcb5d received from {username='slave'} at 127.0.57.194:59197 I0507 06:28:41.223024 3089 tablet_copy_service.cc:413] P 9b897477ac24457ba3057221142ea9b4: ending tablet copy session 1f3225ac14a24ce4be60bcf2c130d634-9bbbc2f7a1fa40c2a9c8edb97b9fcb5d on tablet 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d with peer 1f3225ac14a24ce4be60bcf2c130d634 I0507 06:28:41.225332 2947 raft_consensus.cc:1126] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 1f3225ac14a24ce4be60bcf2c130d634 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[] I0507 06:28:41.242357 3154 raft_consensus.cc:472] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 0c1b3bd57dab421faa01448383625df1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9b897477ac24457ba3057221142ea9b4) I0507 06:28:41.242535 3154 raft_consensus.cc:494] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 0c1b3bd57dab421faa01448383625df1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9b897477ac24457ba3057221142ea9b4" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 38257 } } peers { permanent_uuid: "0c1b3bd57dab421faa01448383625df1" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 35893 } } peers { permanent_uuid: "1f3225ac14a24ce4be60bcf2c130d634" member_type: NON_VOTER last_known_addr { host: "127.0.57.194" port: 41551 } attrs { } } I0507 06:28:41.243083 3154 leader_election.cc:296] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 0c1b3bd57dab421faa01448383625df1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9b897477ac24457ba3057221142ea9b4 (127.0.57.195:38257) I0507 06:28:41.335520 3154 raft_consensus.cc:472] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 0c1b3bd57dab421faa01448383625df1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9b897477ac24457ba3057221142ea9b4) I0507 06:28:41.335695 3154 raft_consensus.cc:494] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 0c1b3bd57dab421faa01448383625df1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9b897477ac24457ba3057221142ea9b4" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 38257 } } peers { permanent_uuid: "0c1b3bd57dab421faa01448383625df1" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 35893 } } peers { permanent_uuid: "1f3225ac14a24ce4be60bcf2c130d634" member_type: NON_VOTER last_known_addr { host: "127.0.57.194" port: 41551 } attrs { } } W0507 06:28:41.336040 2748 leader_election.cc:341] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 0c1b3bd57dab421faa01448383625df1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9b897477ac24457ba3057221142ea9b4 (127.0.57.195:38257): Timed out: RequestConsensusVote RPC to 127.0.57.195:38257 timed out after 0.092s (SENT) I0507 06:28:41.336133 2748 leader_election.cc:310] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 0c1b3bd57dab421faa01448383625df1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 0c1b3bd57dab421faa01448383625df1; no voters: 9b897477ac24457ba3057221142ea9b4 I0507 06:28:41.336288 3154 leader_election.cc:296] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 0c1b3bd57dab421faa01448383625df1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9b897477ac24457ba3057221142ea9b4 (127.0.57.195:38257) I0507 06:28:41.336544 3154 raft_consensus.cc:2588] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 0c1b3bd57dab421faa01448383625df1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority I0507 06:28:41.410639 3165 raft_consensus.cc:472] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 0c1b3bd57dab421faa01448383625df1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9b897477ac24457ba3057221142ea9b4) I0507 06:28:41.410863 3165 raft_consensus.cc:494] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 0c1b3bd57dab421faa01448383625df1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9b897477ac24457ba3057221142ea9b4" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 38257 } } peers { permanent_uuid: "0c1b3bd57dab421faa01448383625df1" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 35893 } } peers { permanent_uuid: "1f3225ac14a24ce4be60bcf2c130d634" member_type: NON_VOTER last_known_addr { host: "127.0.57.194" port: 41551 } attrs { } } I0507 06:28:41.411526 3165 leader_election.cc:296] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 0c1b3bd57dab421faa01448383625df1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9b897477ac24457ba3057221142ea9b4 (127.0.57.195:38257) W0507 06:28:41.429344 2748 leader_election.cc:341] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 0c1b3bd57dab421faa01448383625df1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9b897477ac24457ba3057221142ea9b4 (127.0.57.195:38257): Timed out: RequestConsensusVote RPC to 127.0.57.195:38257 timed out after 0.092s (SENT) I0507 06:28:41.429504 2748 leader_election.cc:310] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 0c1b3bd57dab421faa01448383625df1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 0c1b3bd57dab421faa01448383625df1; no voters: 9b897477ac24457ba3057221142ea9b4 I0507 06:28:41.429920 3165 raft_consensus.cc:2588] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 0c1b3bd57dab421faa01448383625df1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority W0507 06:28:41.476431 2748 leader_election.cc:341] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 0c1b3bd57dab421faa01448383625df1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9b897477ac24457ba3057221142ea9b4 (127.0.57.195:38257): Timed out: RequestConsensusVote RPC to 127.0.57.195:38257 timed out after 0.064s (SENT) I0507 06:28:41.476569 2748 leader_election.cc:310] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 0c1b3bd57dab421faa01448383625df1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 0c1b3bd57dab421faa01448383625df1; no voters: 9b897477ac24457ba3057221142ea9b4 I0507 06:28:41.477015 3165 raft_consensus.cc:2588] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 0c1b3bd57dab421faa01448383625df1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority I0507 06:28:41.569391 3165 raft_consensus.cc:472] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 0c1b3bd57dab421faa01448383625df1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9b897477ac24457ba3057221142ea9b4) I0507 06:28:41.569593 3165 raft_consensus.cc:494] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 0c1b3bd57dab421faa01448383625df1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9b897477ac24457ba3057221142ea9b4" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 38257 } } peers { permanent_uuid: "0c1b3bd57dab421faa01448383625df1" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 35893 } } peers { permanent_uuid: "1f3225ac14a24ce4be60bcf2c130d634" member_type: NON_VOTER last_known_addr { host: "127.0.57.194" port: 41551 } attrs { } } I0507 06:28:41.570179 3165 leader_election.cc:296] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 0c1b3bd57dab421faa01448383625df1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9b897477ac24457ba3057221142ea9b4 (127.0.57.195:38257) I0507 06:28:41.645758 3165 raft_consensus.cc:472] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 0c1b3bd57dab421faa01448383625df1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9b897477ac24457ba3057221142ea9b4) W0507 06:28:41.646180 2748 leader_election.cc:341] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 0c1b3bd57dab421faa01448383625df1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9b897477ac24457ba3057221142ea9b4 (127.0.57.195:38257): Timed out: RequestConsensusVote RPC to 127.0.57.195:38257 timed out after 0.075s (SENT) I0507 06:28:41.645952 3165 raft_consensus.cc:494] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 0c1b3bd57dab421faa01448383625df1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9b897477ac24457ba3057221142ea9b4" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 38257 } } peers { permanent_uuid: "0c1b3bd57dab421faa01448383625df1" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 35893 } } peers { permanent_uuid: "1f3225ac14a24ce4be60bcf2c130d634" member_type: NON_VOTER last_known_addr { host: "127.0.57.194" port: 41551 } attrs { } } I0507 06:28:41.646313 2748 leader_election.cc:310] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 0c1b3bd57dab421faa01448383625df1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 0c1b3bd57dab421faa01448383625df1; no voters: 9b897477ac24457ba3057221142ea9b4 I0507 06:28:41.646644 3154 raft_consensus.cc:2588] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 0c1b3bd57dab421faa01448383625df1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority I0507 06:28:41.646706 3165 leader_election.cc:296] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 0c1b3bd57dab421faa01448383625df1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9b897477ac24457ba3057221142ea9b4 (127.0.57.195:38257) I0507 06:28:41.946635 3165 raft_consensus.cc:472] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 0c1b3bd57dab421faa01448383625df1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9b897477ac24457ba3057221142ea9b4) I0507 06:28:41.946841 3165 raft_consensus.cc:494] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 0c1b3bd57dab421faa01448383625df1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9b897477ac24457ba3057221142ea9b4" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 38257 } } peers { permanent_uuid: "0c1b3bd57dab421faa01448383625df1" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 35893 } } peers { permanent_uuid: "1f3225ac14a24ce4be60bcf2c130d634" member_type: NON_VOTER last_known_addr { host: "127.0.57.194" port: 41551 } attrs { } } I0507 06:28:41.947432 3165 leader_election.cc:296] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 0c1b3bd57dab421faa01448383625df1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9b897477ac24457ba3057221142ea9b4 (127.0.57.195:38257) W0507 06:28:41.967983 2748 leader_election.cc:341] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 0c1b3bd57dab421faa01448383625df1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9b897477ac24457ba3057221142ea9b4 (127.0.57.195:38257): Timed out: RequestConsensusVote RPC to 127.0.57.195:38257 timed out after 0.321s (SENT) I0507 06:28:41.968106 2748 leader_election.cc:310] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 0c1b3bd57dab421faa01448383625df1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 0c1b3bd57dab421faa01448383625df1; no voters: 9b897477ac24457ba3057221142ea9b4 I0507 06:28:41.968516 3165 raft_consensus.cc:2588] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 0c1b3bd57dab421faa01448383625df1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority I0507 06:28:42.121578 3165 raft_consensus.cc:472] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 0c1b3bd57dab421faa01448383625df1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9b897477ac24457ba3057221142ea9b4) I0507 06:28:42.121794 3165 raft_consensus.cc:494] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 0c1b3bd57dab421faa01448383625df1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9b897477ac24457ba3057221142ea9b4" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 38257 } } peers { permanent_uuid: "0c1b3bd57dab421faa01448383625df1" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 35893 } } peers { permanent_uuid: "1f3225ac14a24ce4be60bcf2c130d634" member_type: NON_VOTER last_known_addr { host: "127.0.57.194" port: 41551 } attrs { } } I0507 06:28:42.122516 3165 leader_election.cc:296] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 0c1b3bd57dab421faa01448383625df1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9b897477ac24457ba3057221142ea9b4 (127.0.57.195:38257) W0507 06:28:42.307672 2748 leader_election.cc:341] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 0c1b3bd57dab421faa01448383625df1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9b897477ac24457ba3057221142ea9b4 (127.0.57.195:38257): Timed out: RequestConsensusVote RPC to 127.0.57.195:38257 timed out after 0.360s (SENT) I0507 06:28:42.307811 2748 leader_election.cc:310] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 0c1b3bd57dab421faa01448383625df1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 0c1b3bd57dab421faa01448383625df1; no voters: 9b897477ac24457ba3057221142ea9b4 I0507 06:28:42.308176 3165 raft_consensus.cc:2588] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 0c1b3bd57dab421faa01448383625df1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority W0507 06:28:42.561058 2748 leader_election.cc:341] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 0c1b3bd57dab421faa01448383625df1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9b897477ac24457ba3057221142ea9b4 (127.0.57.195:38257): Timed out: RequestConsensusVote RPC to 127.0.57.195:38257 timed out after 0.438s (SENT) I0507 06:28:42.561172 2748 leader_election.cc:310] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 0c1b3bd57dab421faa01448383625df1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 0c1b3bd57dab421faa01448383625df1; no voters: 9b897477ac24457ba3057221142ea9b4 I0507 06:28:42.561514 3165 raft_consensus.cc:2588] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 0c1b3bd57dab421faa01448383625df1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority I0507 06:28:43.176447 3166 raft_consensus.cc:472] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 0c1b3bd57dab421faa01448383625df1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9b897477ac24457ba3057221142ea9b4) I0507 06:28:43.176677 3166 raft_consensus.cc:494] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 0c1b3bd57dab421faa01448383625df1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9b897477ac24457ba3057221142ea9b4" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 38257 } } peers { permanent_uuid: "0c1b3bd57dab421faa01448383625df1" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 35893 } } peers { permanent_uuid: "1f3225ac14a24ce4be60bcf2c130d634" member_type: NON_VOTER last_known_addr { host: "127.0.57.194" port: 41551 } attrs { } } I0507 06:28:43.177237 3166 leader_election.cc:296] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 0c1b3bd57dab421faa01448383625df1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9b897477ac24457ba3057221142ea9b4 (127.0.57.195:38257) I0507 06:28:44.648355 3167 raft_consensus.cc:472] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 0c1b3bd57dab421faa01448383625df1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9b897477ac24457ba3057221142ea9b4) I0507 06:28:44.648586 3167 raft_consensus.cc:494] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 0c1b3bd57dab421faa01448383625df1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9b897477ac24457ba3057221142ea9b4" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 38257 } } peers { permanent_uuid: "0c1b3bd57dab421faa01448383625df1" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 35893 } } peers { permanent_uuid: "1f3225ac14a24ce4be60bcf2c130d634" member_type: NON_VOTER last_known_addr { host: "127.0.57.194" port: 41551 } attrs { } } W0507 06:28:44.648826 2748 leader_election.cc:341] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 0c1b3bd57dab421faa01448383625df1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9b897477ac24457ba3057221142ea9b4 (127.0.57.195:38257): Timed out: RequestConsensusVote RPC to 127.0.57.195:38257 timed out after 1.470s (SENT) I0507 06:28:44.648983 2748 leader_election.cc:310] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 0c1b3bd57dab421faa01448383625df1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 0c1b3bd57dab421faa01448383625df1; no voters: 9b897477ac24457ba3057221142ea9b4 I0507 06:28:44.649263 3167 leader_election.cc:296] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 0c1b3bd57dab421faa01448383625df1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9b897477ac24457ba3057221142ea9b4 (127.0.57.195:38257) I0507 06:28:44.649498 3167 raft_consensus.cc:2588] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 0c1b3bd57dab421faa01448383625df1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority W0507 06:28:45.011308 2748 leader_election.cc:341] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 0c1b3bd57dab421faa01448383625df1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9b897477ac24457ba3057221142ea9b4 (127.0.57.195:38257): Timed out: RequestConsensusVote RPC to 127.0.57.195:38257 timed out after 0.361s (SENT) I0507 06:28:45.011456 2748 leader_election.cc:310] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 0c1b3bd57dab421faa01448383625df1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 0c1b3bd57dab421faa01448383625df1; no voters: 9b897477ac24457ba3057221142ea9b4 I0507 06:28:45.012025 3167 raft_consensus.cc:2588] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 0c1b3bd57dab421faa01448383625df1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority I0507 06:28:45.982620 3169 raft_consensus.cc:472] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 0c1b3bd57dab421faa01448383625df1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9b897477ac24457ba3057221142ea9b4) I0507 06:28:45.982901 3169 raft_consensus.cc:494] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 0c1b3bd57dab421faa01448383625df1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9b897477ac24457ba3057221142ea9b4" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 38257 } } peers { permanent_uuid: "0c1b3bd57dab421faa01448383625df1" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 35893 } } peers { permanent_uuid: "1f3225ac14a24ce4be60bcf2c130d634" member_type: NON_VOTER last_known_addr { host: "127.0.57.194" port: 41551 } attrs { } } I0507 06:28:45.983587 3169 leader_election.cc:296] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 0c1b3bd57dab421faa01448383625df1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9b897477ac24457ba3057221142ea9b4 (127.0.57.195:38257) I0507 06:28:47.470021 3170 raft_consensus.cc:472] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 0c1b3bd57dab421faa01448383625df1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9b897477ac24457ba3057221142ea9b4) I0507 06:28:47.470225 3170 raft_consensus.cc:494] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 0c1b3bd57dab421faa01448383625df1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9b897477ac24457ba3057221142ea9b4" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 38257 } } peers { permanent_uuid: "0c1b3bd57dab421faa01448383625df1" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 35893 } } peers { permanent_uuid: "1f3225ac14a24ce4be60bcf2c130d634" member_type: NON_VOTER last_known_addr { host: "127.0.57.194" port: 41551 } attrs { } } I0507 06:28:47.470916 3170 leader_election.cc:296] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 0c1b3bd57dab421faa01448383625df1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9b897477ac24457ba3057221142ea9b4 (127.0.57.195:38257) W0507 06:28:47.471213 2748 leader_election.cc:341] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 0c1b3bd57dab421faa01448383625df1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9b897477ac24457ba3057221142ea9b4 (127.0.57.195:38257): Timed out: RequestConsensusVote RPC to 127.0.57.195:38257 timed out after 1.486s (SENT) I0507 06:28:47.471310 2748 leader_election.cc:310] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 0c1b3bd57dab421faa01448383625df1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 0c1b3bd57dab421faa01448383625df1; no voters: 9b897477ac24457ba3057221142ea9b4 I0507 06:28:47.471732 3170 raft_consensus.cc:2588] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 0c1b3bd57dab421faa01448383625df1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority W0507 06:28:48.135233 2748 leader_election.cc:341] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 0c1b3bd57dab421faa01448383625df1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9b897477ac24457ba3057221142ea9b4 (127.0.57.195:38257): Timed out: RequestConsensusVote RPC to 127.0.57.195:38257 timed out after 0.663s (SENT) I0507 06:28:48.135403 2748 leader_election.cc:310] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 0c1b3bd57dab421faa01448383625df1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 0c1b3bd57dab421faa01448383625df1; no voters: 9b897477ac24457ba3057221142ea9b4 I0507 06:28:48.136242 3171 raft_consensus.cc:2588] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 0c1b3bd57dab421faa01448383625df1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority I0507 06:28:48.804329 3069 tablet_service.cc:1286] Received RequestConsensusVote() RPC: tablet_id: "9bbbc2f7a1fa40c2a9c8edb97b9fcb5d" candidate_uuid: "0c1b3bd57dab421faa01448383625df1" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "9b897477ac24457ba3057221142ea9b4" is_pre_election: true I0507 06:28:48.804582 3069 raft_consensus.cc:2307] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 9b897477ac24457ba3057221142ea9b4 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate 0c1b3bd57dab421faa01448383625df1 for term 2 because replica is either leader or believes a valid leader to be alive. I0507 06:28:48.806275 3066 tablet_service.cc:1286] Received RequestConsensusVote() RPC: tablet_id: "9bbbc2f7a1fa40c2a9c8edb97b9fcb5d" candidate_uuid: "0c1b3bd57dab421faa01448383625df1" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "9b897477ac24457ba3057221142ea9b4" is_pre_election: true I0507 06:28:48.807884 3068 tablet_service.cc:1286] Received RequestConsensusVote() RPC: tablet_id: "9bbbc2f7a1fa40c2a9c8edb97b9fcb5d" candidate_uuid: "0c1b3bd57dab421faa01448383625df1" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "9b897477ac24457ba3057221142ea9b4" is_pre_election: true I0507 06:28:48.807103 3057 tablet_service.cc:1286] Received RequestConsensusVote() RPC: tablet_id: "9bbbc2f7a1fa40c2a9c8edb97b9fcb5d" candidate_uuid: "0c1b3bd57dab421faa01448383625df1" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "9b897477ac24457ba3057221142ea9b4" is_pre_election: true I0507 06:28:48.811313 3066 raft_consensus.cc:2307] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 9b897477ac24457ba3057221142ea9b4 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate 0c1b3bd57dab421faa01448383625df1 for term 2 because replica is either leader or believes a valid leader to be alive. I0507 06:28:48.808591 3061 tablet_service.cc:1286] Received RequestConsensusVote() RPC: tablet_id: "9bbbc2f7a1fa40c2a9c8edb97b9fcb5d" candidate_uuid: "0c1b3bd57dab421faa01448383625df1" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "9b897477ac24457ba3057221142ea9b4" is_pre_election: true I0507 06:28:48.811434 3067 tablet_service.cc:1286] Received RequestConsensusVote() RPC: tablet_id: "9bbbc2f7a1fa40c2a9c8edb97b9fcb5d" candidate_uuid: "0c1b3bd57dab421faa01448383625df1" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "9b897477ac24457ba3057221142ea9b4" is_pre_election: true I0507 06:28:48.812310 3065 tablet_service.cc:1286] Received RequestConsensusVote() RPC: tablet_id: "9bbbc2f7a1fa40c2a9c8edb97b9fcb5d" candidate_uuid: "0c1b3bd57dab421faa01448383625df1" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "9b897477ac24457ba3057221142ea9b4" is_pre_election: true I0507 06:28:48.812805 3055 tablet_service.cc:1286] Received RequestConsensusVote() RPC: tablet_id: "9bbbc2f7a1fa40c2a9c8edb97b9fcb5d" candidate_uuid: "0c1b3bd57dab421faa01448383625df1" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "9b897477ac24457ba3057221142ea9b4" is_pre_election: true I0507 06:28:48.810653 3054 tablet_service.cc:1286] Received RequestConsensusVote() RPC: tablet_id: "9bbbc2f7a1fa40c2a9c8edb97b9fcb5d" candidate_uuid: "0c1b3bd57dab421faa01448383625df1" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "9b897477ac24457ba3057221142ea9b4" is_pre_election: true I0507 06:28:48.813019 3059 tablet_service.cc:1286] Received RequestConsensusVote() RPC: tablet_id: "9bbbc2f7a1fa40c2a9c8edb97b9fcb5d" candidate_uuid: "0c1b3bd57dab421faa01448383625df1" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "9b897477ac24457ba3057221142ea9b4" is_pre_election: true I0507 06:28:48.813045 3063 tablet_service.cc:1286] Received RequestConsensusVote() RPC: tablet_id: "9bbbc2f7a1fa40c2a9c8edb97b9fcb5d" candidate_uuid: "0c1b3bd57dab421faa01448383625df1" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "9b897477ac24457ba3057221142ea9b4" is_pre_election: true I0507 06:28:48.817191 3061 raft_consensus.cc:2307] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 9b897477ac24457ba3057221142ea9b4 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate 0c1b3bd57dab421faa01448383625df1 for term 2 because replica is either leader or believes a valid leader to be alive. I0507 06:28:48.817622 3057 raft_consensus.cc:2321] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 9b897477ac24457ba3057221142ea9b4 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate 0c1b3bd57dab421faa01448383625df1 for term 2 because replica is already servicing an update from a current leader or another vote. I0507 06:28:48.818579 3068 raft_consensus.cc:2321] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 9b897477ac24457ba3057221142ea9b4 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate 0c1b3bd57dab421faa01448383625df1 for term 2 because replica is already servicing an update from a current leader or another vote. I0507 06:28:48.819371 3055 raft_consensus.cc:2321] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 9b897477ac24457ba3057221142ea9b4 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate 0c1b3bd57dab421faa01448383625df1 for term 2 because replica is already servicing an update from a current leader or another vote. I0507 06:28:48.819659 3054 raft_consensus.cc:2321] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 9b897477ac24457ba3057221142ea9b4 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate 0c1b3bd57dab421faa01448383625df1 for term 2 because replica is already servicing an update from a current leader or another vote. I0507 06:28:48.820022 3067 raft_consensus.cc:2321] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 9b897477ac24457ba3057221142ea9b4 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate 0c1b3bd57dab421faa01448383625df1 for term 2 because replica is already servicing an update from a current leader or another vote. I0507 06:28:48.821477 3065 raft_consensus.cc:2321] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 9b897477ac24457ba3057221142ea9b4 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate 0c1b3bd57dab421faa01448383625df1 for term 2 because replica is already servicing an update from a current leader or another vote. W0507 06:28:48.821467 2864 generic_service.cc:128] {username='slave'} at 127.0.0.1:57758 forcing change of non-runtime-safe flag enable_leader_failure_detection I0507 06:28:48.821576 2864 generic_service.cc:148] {username='slave'} at 127.0.0.1:57758 changed flags via RPC: enable_leader_failure_detection from 'true' to 'false' I0507 06:28:48.822324 3059 raft_consensus.cc:2321] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 9b897477ac24457ba3057221142ea9b4 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate 0c1b3bd57dab421faa01448383625df1 for term 2 because replica is already servicing an update from a current leader or another vote. I0507 06:28:48.822543 3063 raft_consensus.cc:2321] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 9b897477ac24457ba3057221142ea9b4 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate 0c1b3bd57dab421faa01448383625df1 for term 2 because replica is already servicing an update from a current leader or another vote. W0507 06:28:48.823316 2987 generic_service.cc:128] {username='slave'} at 127.0.0.1:39508 forcing change of non-runtime-safe flag enable_leader_failure_detection I0507 06:28:48.823444 2987 generic_service.cc:148] {username='slave'} at 127.0.0.1:39508 changed flags via RPC: enable_leader_failure_detection from 'true' to 'false' W0507 06:28:48.824934 3109 generic_service.cc:128] {username='slave'} at 127.0.0.1:60272 forcing change of non-runtime-safe flag enable_leader_failure_detection I0507 06:28:48.825026 3109 generic_service.cc:148] {username='slave'} at 127.0.0.1:60272 changed flags via RPC: enable_leader_failure_detection from 'true' to 'false' I0507 06:28:48.828474 231 external_mini_cluster.cc:1172] Attempting to check leaks for /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver pid 2871 I0507 06:28:48.829337 2987 generic_service.cc:171] Checking for leaks (request via RPC) I0507 06:28:48.868569 3176 raft_consensus.cc:472] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 0c1b3bd57dab421faa01448383625df1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9b897477ac24457ba3057221142ea9b4) I0507 06:28:48.868743 3176 raft_consensus.cc:494] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 0c1b3bd57dab421faa01448383625df1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9b897477ac24457ba3057221142ea9b4" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 38257 } } peers { permanent_uuid: "0c1b3bd57dab421faa01448383625df1" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 35893 } } peers { permanent_uuid: "1f3225ac14a24ce4be60bcf2c130d634" member_type: NON_VOTER last_known_addr { host: "127.0.57.194" port: 41551 } attrs { } } I0507 06:28:48.869850 3176 leader_election.cc:296] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 0c1b3bd57dab421faa01448383625df1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9b897477ac24457ba3057221142ea9b4 (127.0.57.195:38257) I0507 06:28:48.870050 3057 tablet_service.cc:1286] Received RequestConsensusVote() RPC: tablet_id: "9bbbc2f7a1fa40c2a9c8edb97b9fcb5d" candidate_uuid: "0c1b3bd57dab421faa01448383625df1" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "9b897477ac24457ba3057221142ea9b4" is_pre_election: true I0507 06:28:48.870272 3057 raft_consensus.cc:2307] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 9b897477ac24457ba3057221142ea9b4 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate 0c1b3bd57dab421faa01448383625df1 for term 2 because replica is either leader or believes a valid leader to be alive. I0507 06:28:48.870808 2748 leader_election.cc:310] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 0c1b3bd57dab421faa01448383625df1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 0c1b3bd57dab421faa01448383625df1; no voters: 9b897477ac24457ba3057221142ea9b4 I0507 06:28:48.871124 3176 raft_consensus.cc:2588] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 0c1b3bd57dab421faa01448383625df1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority I0507 06:28:48.878170 231 external_mini_cluster.cc:1131] Killing /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver with pid 2871 I0507 06:28:48.889612 231 external_mini_cluster.cc:933] Running /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.NonVoterReplicasDoNotVote.1557210500371863-231/raft_consensus-itest-cluster/ts-1/wal --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.NonVoterReplicasDoNotVote.1557210500371863-231/raft_consensus-itest-cluster/ts-1/data --block_manager=log --rpc_bind_addresses=127.0.57.194:41551 --local_ip_for_outbound_sockets=127.0.57.194 --webserver_port=36339 --webserver_interface=127.0.57.194 --tserver_master_addrs=127.0.57.254:46409 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.NonVoterReplicasDoNotVote.1557210500371863-231/raft_consensus-itest-cluster/ts-1/logs --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.NonVoterReplicasDoNotVote.1557210500371863-231/raft_consensus-itest-cluster/ts-1/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_heartbeat_interval_ms=64 --leader_failure_max_missed_heartbeat_periods=1 with env {} I0507 06:28:48.920166 3176 raft_consensus.cc:472] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 0c1b3bd57dab421faa01448383625df1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9b897477ac24457ba3057221142ea9b4) I0507 06:28:48.920465 3176 raft_consensus.cc:494] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 0c1b3bd57dab421faa01448383625df1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9b897477ac24457ba3057221142ea9b4" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 38257 } } peers { permanent_uuid: "0c1b3bd57dab421faa01448383625df1" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 35893 } } peers { permanent_uuid: "1f3225ac14a24ce4be60bcf2c130d634" member_type: NON_VOTER last_known_addr { host: "127.0.57.194" port: 41551 } attrs { } } I0507 06:28:48.921093 3176 leader_election.cc:296] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 0c1b3bd57dab421faa01448383625df1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9b897477ac24457ba3057221142ea9b4 (127.0.57.195:38257) I0507 06:28:48.921916 3057 tablet_service.cc:1286] Received RequestConsensusVote() RPC: tablet_id: "9bbbc2f7a1fa40c2a9c8edb97b9fcb5d" candidate_uuid: "0c1b3bd57dab421faa01448383625df1" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "9b897477ac24457ba3057221142ea9b4" is_pre_election: true I0507 06:28:48.922096 3057 raft_consensus.cc:2307] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 9b897477ac24457ba3057221142ea9b4 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate 0c1b3bd57dab421faa01448383625df1 for term 2 because replica is either leader or believes a valid leader to be alive. I0507 06:28:48.922508 2748 leader_election.cc:310] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 0c1b3bd57dab421faa01448383625df1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 0c1b3bd57dab421faa01448383625df1; no voters: 9b897477ac24457ba3057221142ea9b4 I0507 06:28:48.922909 3176 raft_consensus.cc:2588] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 0c1b3bd57dab421faa01448383625df1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority W0507 06:28:48.932727 3001 consensus_peers.cc:457] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 9b897477ac24457ba3057221142ea9b4 -> Peer 1f3225ac14a24ce4be60bcf2c130d634 (127.0.57.194:41551): Couldn't send request to peer 1f3225ac14a24ce4be60bcf2c130d634. Status: Network error: Client connection negotiation failed: client connection to 127.0.57.194:41551: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry. I0507 06:28:48.985877 3176 raft_consensus.cc:472] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 0c1b3bd57dab421faa01448383625df1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9b897477ac24457ba3057221142ea9b4) I0507 06:28:48.986068 3176 raft_consensus.cc:494] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 0c1b3bd57dab421faa01448383625df1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9b897477ac24457ba3057221142ea9b4" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 38257 } } peers { permanent_uuid: "0c1b3bd57dab421faa01448383625df1" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 35893 } } peers { permanent_uuid: "1f3225ac14a24ce4be60bcf2c130d634" member_type: NON_VOTER last_known_addr { host: "127.0.57.194" port: 41551 } attrs { } } I0507 06:28:48.986817 3176 leader_election.cc:296] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 0c1b3bd57dab421faa01448383625df1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9b897477ac24457ba3057221142ea9b4 (127.0.57.195:38257) I0507 06:28:48.987720 3057 tablet_service.cc:1286] Received RequestConsensusVote() RPC: tablet_id: "9bbbc2f7a1fa40c2a9c8edb97b9fcb5d" candidate_uuid: "0c1b3bd57dab421faa01448383625df1" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "9b897477ac24457ba3057221142ea9b4" is_pre_election: true I0507 06:28:48.987963 3057 raft_consensus.cc:2307] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 9b897477ac24457ba3057221142ea9b4 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate 0c1b3bd57dab421faa01448383625df1 for term 2 because replica is either leader or believes a valid leader to be alive. I0507 06:28:48.988628 2748 leader_election.cc:310] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 0c1b3bd57dab421faa01448383625df1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 0c1b3bd57dab421faa01448383625df1; no voters: 9b897477ac24457ba3057221142ea9b4 I0507 06:28:48.988966 3176 raft_consensus.cc:2588] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 0c1b3bd57dab421faa01448383625df1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority I0507 06:28:49.040665 3176 raft_consensus.cc:472] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 0c1b3bd57dab421faa01448383625df1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9b897477ac24457ba3057221142ea9b4) I0507 06:28:49.040902 3176 raft_consensus.cc:494] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 0c1b3bd57dab421faa01448383625df1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9b897477ac24457ba3057221142ea9b4" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 38257 } } peers { permanent_uuid: "0c1b3bd57dab421faa01448383625df1" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 35893 } } peers { permanent_uuid: "1f3225ac14a24ce4be60bcf2c130d634" member_type: NON_VOTER last_known_addr { host: "127.0.57.194" port: 41551 } attrs { } } I0507 06:28:49.041990 3176 leader_election.cc:296] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 0c1b3bd57dab421faa01448383625df1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9b897477ac24457ba3057221142ea9b4 (127.0.57.195:38257) I0507 06:28:49.042536 3057 tablet_service.cc:1286] Received RequestConsensusVote() RPC: tablet_id: "9bbbc2f7a1fa40c2a9c8edb97b9fcb5d" candidate_uuid: "0c1b3bd57dab421faa01448383625df1" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "9b897477ac24457ba3057221142ea9b4" is_pre_election: true I0507 06:28:49.042783 3057 raft_consensus.cc:2307] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 9b897477ac24457ba3057221142ea9b4 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate 0c1b3bd57dab421faa01448383625df1 for term 2 because replica is either leader or believes a valid leader to be alive. I0507 06:28:49.043337 2748 leader_election.cc:310] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 0c1b3bd57dab421faa01448383625df1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 0c1b3bd57dab421faa01448383625df1; no voters: 9b897477ac24457ba3057221142ea9b4 I0507 06:28:49.043732 3176 raft_consensus.cc:2588] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 0c1b3bd57dab421faa01448383625df1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority WARNING: Logging before InitGoogleLogging() is written to STDERR W0507 06:28:49.066840 3177 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0507 06:28:49.067077 3177 flags.cc:404] Enabled unsafe flag: --never_fsync=true W0507 06:28:49.069327 3177 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W0507 06:28:49.069435 3177 flags.cc:404] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.57.194 I0507 06:28:49.073858 3177 tablet_server_main.cc:78] Tablet server non-default flags: --leader_failure_max_missed_heartbeat_periods=1 --raft_heartbeat_interval_ms=64 --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.NonVoterReplicasDoNotVote.1557210500371863-231/raft_consensus-itest-cluster/ts-1/data --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.NonVoterReplicasDoNotVote.1557210500371863-231/raft_consensus-itest-cluster/ts-1/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.0.57.194:41551 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.NonVoterReplicasDoNotVote.1557210500371863-231/raft_consensus-itest-cluster/ts-1/data/info.pb --webserver_interface=127.0.57.194 --webserver_port=36339 --tserver_master_addrs=127.0.57.254:46409 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.0.57.194 --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.NonVoterReplicasDoNotVote.1557210500371863-231/raft_consensus-itest-cluster/ts-1/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.10.0-cdh6.x-SNAPSHOT revision 7d0ee26697926e87af9ef395008799577f6ab43f build type FASTDEBUG built by jenkins at 06 May 2019 22:44:21 PST on quasar-knmpfx-1.vpc.cloudera.com build id 322017 ASAN enabled I0507 06:28:49.082895 3177 system_ntp.cc:202] NTP initialized. Skew: 500ppm Current error: 39016us I0507 06:28:49.083660 3177 fs_manager.cc:263] Metadata directory not provided I0507 06:28:49.083762 3177 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0507 06:28:49.091095 3176 raft_consensus.cc:472] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 0c1b3bd57dab421faa01448383625df1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9b897477ac24457ba3057221142ea9b4) I0507 06:28:49.091296 3177 fs_manager.cc:399] Time spent opening directory manager: real 0.006s user 0.003s sys 0.001s I0507 06:28:49.091246 3176 raft_consensus.cc:494] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 0c1b3bd57dab421faa01448383625df1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9b897477ac24457ba3057221142ea9b4" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 38257 } } peers { permanent_uuid: "0c1b3bd57dab421faa01448383625df1" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 35893 } } peers { permanent_uuid: "1f3225ac14a24ce4be60bcf2c130d634" member_type: NON_VOTER last_known_addr { host: "127.0.57.194" port: 41551 } attrs { } } I0507 06:28:49.091766 3176 leader_election.cc:296] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 0c1b3bd57dab421faa01448383625df1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9b897477ac24457ba3057221142ea9b4 (127.0.57.195:38257) I0507 06:28:49.092033 3177 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0507 06:28:49.092645 3057 tablet_service.cc:1286] Received RequestConsensusVote() RPC: tablet_id: "9bbbc2f7a1fa40c2a9c8edb97b9fcb5d" candidate_uuid: "0c1b3bd57dab421faa01448383625df1" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "9b897477ac24457ba3057221142ea9b4" is_pre_election: true I0507 06:28:49.092823 3177 file_cache.cc:466] Constructed file cache lbm with capacity 419430 I0507 06:28:49.092900 3057 raft_consensus.cc:2307] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 9b897477ac24457ba3057221142ea9b4 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate 0c1b3bd57dab421faa01448383625df1 for term 2 because replica is either leader or believes a valid leader to be alive. I0507 06:28:49.093456 2748 leader_election.cc:310] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 0c1b3bd57dab421faa01448383625df1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 0c1b3bd57dab421faa01448383625df1; no voters: 9b897477ac24457ba3057221142ea9b4 I0507 06:28:49.093835 3176 raft_consensus.cc:2588] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 0c1b3bd57dab421faa01448383625df1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority I0507 06:28:49.096140 3177 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s I0507 06:28:49.096222 3177 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.NonVoterReplicasDoNotVote.1557210500371863-231/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.NonVoterReplicasDoNotVote.1557210500371863-231/raft_consensus-itest-cluster/ts-1/wal uuid: "1f3225ac14a24ce4be60bcf2c130d634" format_stamp: "Formatted at 2019-05-07 06:28:38 on dist-test-slave-dist-test-slave-rqp6" I0507 06:28:49.097275 3177 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.NonVoterReplicasDoNotVote.1557210500371863-231/raft_consensus-itest-cluster/ts-1/wal metadata directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.NonVoterReplicasDoNotVote.1557210500371863-231/raft_consensus-itest-cluster/ts-1/wal 1 data directories: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.NonVoterReplicasDoNotVote.1557210500371863-231/raft_consensus-itest-cluster/ts-1/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0507 06:28:49.134910 3177 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0507 06:28:49.137081 3177 ts_tablet_manager.cc:344] Loading tablet metadata (0/1 complete) I0507 06:28:49.141585 3177 ts_tablet_manager.cc:356] Loaded tablet metadata (1 total tablets, 1 live tablets) I0507 06:28:49.141664 3177 ts_tablet_manager.cc:362] Registering tablets (0/1 complete) I0507 06:28:49.145061 3193 tablet_bootstrap.cc:439] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 1f3225ac14a24ce4be60bcf2c130d634: Bootstrap starting. I0507 06:28:49.148614 3177 ts_tablet_manager.cc:376] Registered 1 tablets I0507 06:28:49.164300 3193 log.cc:525] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 1f3225ac14a24ce4be60bcf2c130d634: Log is configured to *not* fsync() on all Append() calls I0507 06:28:49.169080 3176 raft_consensus.cc:472] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 0c1b3bd57dab421faa01448383625df1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9b897477ac24457ba3057221142ea9b4) I0507 06:28:49.169219 3176 raft_consensus.cc:494] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 0c1b3bd57dab421faa01448383625df1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9b897477ac24457ba3057221142ea9b4" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 38257 } } peers { permanent_uuid: "0c1b3bd57dab421faa01448383625df1" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 35893 } } peers { permanent_uuid: "1f3225ac14a24ce4be60bcf2c130d634" member_type: NON_VOTER last_known_addr { host: "127.0.57.194" port: 41551 } attrs { } } I0507 06:28:49.169409 3193 tablet_bootstrap.cc:439] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 1f3225ac14a24ce4be60bcf2c130d634: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I0507 06:28:49.169729 3176 leader_election.cc:296] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 0c1b3bd57dab421faa01448383625df1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9b897477ac24457ba3057221142ea9b4 (127.0.57.195:38257) I0507 06:28:49.169858 3193 tablet_bootstrap.cc:439] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 1f3225ac14a24ce4be60bcf2c130d634: Bootstrap complete. I0507 06:28:49.170281 3057 tablet_service.cc:1286] Received RequestConsensusVote() RPC: tablet_id: "9bbbc2f7a1fa40c2a9c8edb97b9fcb5d" candidate_uuid: "0c1b3bd57dab421faa01448383625df1" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "9b897477ac24457ba3057221142ea9b4" is_pre_election: true I0507 06:28:49.170496 3057 raft_consensus.cc:2307] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 9b897477ac24457ba3057221142ea9b4 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate 0c1b3bd57dab421faa01448383625df1 for term 2 because replica is either leader or believes a valid leader to be alive. I0507 06:28:49.170718 3193 ts_tablet_manager.cc:1053] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 1f3225ac14a24ce4be60bcf2c130d634: Time spent bootstrapping tablet: real 0.026s user 0.004s sys 0.014s I0507 06:28:49.170933 2748 leader_election.cc:310] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 0c1b3bd57dab421faa01448383625df1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 0c1b3bd57dab421faa01448383625df1; no voters: 9b897477ac24457ba3057221142ea9b4 I0507 06:28:49.171205 3176 raft_consensus.cc:2588] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 0c1b3bd57dab421faa01448383625df1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority I0507 06:28:49.181385 3193 raft_consensus.cc:340] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 1f3225ac14a24ce4be60bcf2c130d634 [term 1 LEARNER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9b897477ac24457ba3057221142ea9b4" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 38257 } } peers { permanent_uuid: "0c1b3bd57dab421faa01448383625df1" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 35893 } } peers { permanent_uuid: "1f3225ac14a24ce4be60bcf2c130d634" member_type: NON_VOTER last_known_addr { host: "127.0.57.194" port: 41551 } attrs { } } I0507 06:28:49.183887 3193 raft_consensus.cc:705] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 1f3225ac14a24ce4be60bcf2c130d634 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1f3225ac14a24ce4be60bcf2c130d634, State: Initialized, Role: LEARNER I0507 06:28:49.184249 3193 consensus_queue.cc:229] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 1f3225ac14a24ce4be60bcf2c130d634 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9b897477ac24457ba3057221142ea9b4" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 38257 } } peers { permanent_uuid: "0c1b3bd57dab421faa01448383625df1" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 35893 } } peers { permanent_uuid: "1f3225ac14a24ce4be60bcf2c130d634" member_type: NON_VOTER last_known_addr { host: "127.0.57.194" port: 41551 } attrs { } } I0507 06:28:49.185745 3193 ts_tablet_manager.cc:1081] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 1f3225ac14a24ce4be60bcf2c130d634: Time spent starting tablet: real 0.015s user 0.010s sys 0.002s I0507 06:28:49.204236 3177 rpc_server.cc:205] RPC server started. Bound to: 127.0.57.194:41551 I0507 06:28:49.205569 3177 webserver.cc:308] Webserver started at http://127.0.57.194:36339/ using document root and password file I0507 06:28:49.206351 3177 server_base.cc:608] Dumped server information to /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.NonVoterReplicasDoNotVote.1557210500371863-231/raft_consensus-itest-cluster/ts-1/data/info.pb I0507 06:28:49.208500 231 external_mini_cluster.cc:995] Started /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver as pid 3177 I0507 06:28:49.216958 3300 heartbeater.cc:345] Connected to a master server at 127.0.57.254:46409 I0507 06:28:49.217257 3300 heartbeater.cc:426] Registering TS with master... I0507 06:28:49.217897 3300 heartbeater.cc:475] Master 127.0.57.254:46409 requested a full tablet report, sending... I0507 06:28:49.219719 2702 ts_manager.cc:159] Re-registered known tserver with Master: 1f3225ac14a24ce4be60bcf2c130d634 (127.0.57.194:41551) I0507 06:28:49.221042 2702 master_service.cc:273] Signed X509 certificate for tserver {username='slave'} at 127.0.57.194:58955 I0507 06:28:49.239080 3176 raft_consensus.cc:472] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 0c1b3bd57dab421faa01448383625df1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9b897477ac24457ba3057221142ea9b4) I0507 06:28:49.239238 3176 raft_consensus.cc:494] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 0c1b3bd57dab421faa01448383625df1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9b897477ac24457ba3057221142ea9b4" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 38257 } } peers { permanent_uuid: "0c1b3bd57dab421faa01448383625df1" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 35893 } } peers { permanent_uuid: "1f3225ac14a24ce4be60bcf2c130d634" member_type: NON_VOTER last_known_addr { host: "127.0.57.194" port: 41551 } attrs { } } I0507 06:28:49.239778 3176 leader_election.cc:296] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 0c1b3bd57dab421faa01448383625df1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9b897477ac24457ba3057221142ea9b4 (127.0.57.195:38257) I0507 06:28:49.240451 3057 tablet_service.cc:1286] Received RequestConsensusVote() RPC: tablet_id: "9bbbc2f7a1fa40c2a9c8edb97b9fcb5d" candidate_uuid: "0c1b3bd57dab421faa01448383625df1" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "9b897477ac24457ba3057221142ea9b4" is_pre_election: true I0507 06:28:49.240710 3057 raft_consensus.cc:2307] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 9b897477ac24457ba3057221142ea9b4 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate 0c1b3bd57dab421faa01448383625df1 for term 2 because replica is either leader or believes a valid leader to be alive. I0507 06:28:49.241179 2748 leader_election.cc:310] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 0c1b3bd57dab421faa01448383625df1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 0c1b3bd57dab421faa01448383625df1; no voters: 9b897477ac24457ba3057221142ea9b4 I0507 06:28:49.241463 3176 raft_consensus.cc:2588] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 0c1b3bd57dab421faa01448383625df1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority I0507 06:28:49.247812 3300 heartbeater.cc:467] Master 127.0.57.254:46409 was elected leader, sending a full tablet report... I0507 06:28:49.296005 3176 raft_consensus.cc:472] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 0c1b3bd57dab421faa01448383625df1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9b897477ac24457ba3057221142ea9b4) I0507 06:28:49.296176 3176 raft_consensus.cc:494] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 0c1b3bd57dab421faa01448383625df1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9b897477ac24457ba3057221142ea9b4" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 38257 } } peers { permanent_uuid: "0c1b3bd57dab421faa01448383625df1" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 35893 } } peers { permanent_uuid: "1f3225ac14a24ce4be60bcf2c130d634" member_type: NON_VOTER last_known_addr { host: "127.0.57.194" port: 41551 } attrs { } } I0507 06:28:49.296785 3176 leader_election.cc:296] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 0c1b3bd57dab421faa01448383625df1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9b897477ac24457ba3057221142ea9b4 (127.0.57.195:38257) I0507 06:28:49.297353 3057 tablet_service.cc:1286] Received RequestConsensusVote() RPC: tablet_id: "9bbbc2f7a1fa40c2a9c8edb97b9fcb5d" candidate_uuid: "0c1b3bd57dab421faa01448383625df1" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "9b897477ac24457ba3057221142ea9b4" is_pre_election: true I0507 06:28:49.297566 3057 raft_consensus.cc:2307] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 9b897477ac24457ba3057221142ea9b4 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate 0c1b3bd57dab421faa01448383625df1 for term 2 because replica is either leader or believes a valid leader to be alive. I0507 06:28:49.297936 2748 leader_election.cc:310] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 0c1b3bd57dab421faa01448383625df1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 0c1b3bd57dab421faa01448383625df1; no voters: 9b897477ac24457ba3057221142ea9b4 I0507 06:28:49.298199 3176 raft_consensus.cc:2588] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 0c1b3bd57dab421faa01448383625df1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority I0507 06:28:49.355069 3176 raft_consensus.cc:472] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 0c1b3bd57dab421faa01448383625df1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9b897477ac24457ba3057221142ea9b4) I0507 06:28:49.355253 3176 raft_consensus.cc:494] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 0c1b3bd57dab421faa01448383625df1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9b897477ac24457ba3057221142ea9b4" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 38257 } } peers { permanent_uuid: "0c1b3bd57dab421faa01448383625df1" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 35893 } } peers { permanent_uuid: "1f3225ac14a24ce4be60bcf2c130d634" member_type: NON_VOTER last_known_addr { host: "127.0.57.194" port: 41551 } attrs { } } I0507 06:28:49.355841 3176 leader_election.cc:296] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 0c1b3bd57dab421faa01448383625df1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9b897477ac24457ba3057221142ea9b4 (127.0.57.195:38257) I0507 06:28:49.356577 3057 tablet_service.cc:1286] Received RequestConsensusVote() RPC: tablet_id: "9bbbc2f7a1fa40c2a9c8edb97b9fcb5d" candidate_uuid: "0c1b3bd57dab421faa01448383625df1" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "9b897477ac24457ba3057221142ea9b4" is_pre_election: true I0507 06:28:49.356784 3057 raft_consensus.cc:2307] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 9b897477ac24457ba3057221142ea9b4 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate 0c1b3bd57dab421faa01448383625df1 for term 2 because replica is either leader or believes a valid leader to be alive. I0507 06:28:49.357250 2748 leader_election.cc:310] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 0c1b3bd57dab421faa01448383625df1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 0c1b3bd57dab421faa01448383625df1; no voters: 9b897477ac24457ba3057221142ea9b4 I0507 06:28:49.357513 3176 raft_consensus.cc:2588] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 0c1b3bd57dab421faa01448383625df1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority I0507 06:28:49.418042 3176 raft_consensus.cc:472] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 0c1b3bd57dab421faa01448383625df1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9b897477ac24457ba3057221142ea9b4) I0507 06:28:49.418231 3176 raft_consensus.cc:494] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 0c1b3bd57dab421faa01448383625df1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9b897477ac24457ba3057221142ea9b4" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 38257 } } peers { permanent_uuid: "0c1b3bd57dab421faa01448383625df1" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 35893 } } peers { permanent_uuid: "1f3225ac14a24ce4be60bcf2c130d634" member_type: NON_VOTER last_known_addr { host: "127.0.57.194" port: 41551 } attrs { } } I0507 06:28:49.418861 3176 leader_election.cc:296] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 0c1b3bd57dab421faa01448383625df1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9b897477ac24457ba3057221142ea9b4 (127.0.57.195:38257) I0507 06:28:49.419677 3057 tablet_service.cc:1286] Received RequestConsensusVote() RPC: tablet_id: "9bbbc2f7a1fa40c2a9c8edb97b9fcb5d" candidate_uuid: "0c1b3bd57dab421faa01448383625df1" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "9b897477ac24457ba3057221142ea9b4" is_pre_election: true I0507 06:28:49.419895 3057 raft_consensus.cc:2307] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 9b897477ac24457ba3057221142ea9b4 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate 0c1b3bd57dab421faa01448383625df1 for term 2 because replica is either leader or believes a valid leader to be alive. I0507 06:28:49.420442 2748 leader_election.cc:310] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 0c1b3bd57dab421faa01448383625df1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 0c1b3bd57dab421faa01448383625df1; no voters: 9b897477ac24457ba3057221142ea9b4 I0507 06:28:49.420845 3176 raft_consensus.cc:2588] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 0c1b3bd57dab421faa01448383625df1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority I0507 06:28:49.426288 3257 tablet_service.cc:1403] Received Run Leader Election RPC: tablet_id: "9bbbc2f7a1fa40c2a9c8edb97b9fcb5d" dest_uuid: "1f3225ac14a24ce4be60bcf2c130d634" from {username='slave'} at 127.0.0.1:39528 I0507 06:28:49.429100 231 external_mini_cluster.cc:1172] Attempting to check leaks for /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver pid 2737 I0507 06:28:49.430918 2864 generic_service.cc:171] Checking for leaks (request via RPC) I0507 06:28:49.485878 3176 raft_consensus.cc:472] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 0c1b3bd57dab421faa01448383625df1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9b897477ac24457ba3057221142ea9b4) I0507 06:28:49.486060 3176 raft_consensus.cc:494] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 0c1b3bd57dab421faa01448383625df1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9b897477ac24457ba3057221142ea9b4" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 38257 } } peers { permanent_uuid: "0c1b3bd57dab421faa01448383625df1" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 35893 } } peers { permanent_uuid: "1f3225ac14a24ce4be60bcf2c130d634" member_type: NON_VOTER last_known_addr { host: "127.0.57.194" port: 41551 } attrs { } } I0507 06:28:49.486649 231 external_mini_cluster.cc:1131] Killing /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver with pid 2737 W0507 06:28:49.497424 3002 consensus_peers.cc:457] T 9bbbc2f7a1fa40c2a9c8edb97b9fcb5d P 9b897477ac24457ba3057221142ea9b4 -> Peer 0c1b3bd57dab421faa01448383625df1 (127.0.57.193:35893): Couldn't send request to peer 0c1b3bd57dab421faa01448383625df1. Status: Network error: recv got EOF from 127.0.57.193:35893 (error 108). This is attempt 1: this message will repeat every 5th retry. I0507 06:28:49.498034 231 external_mini_cluster.cc:1172] Attempting to check leaks for /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver pid 3177 I0507 06:28:49.499183 3297 generic_service.cc:171] Checking for leaks (request via RPC) I0507 06:28:49.547761 231 external_mini_cluster.cc:1131] Killing /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver with pid 3177 I0507 06:28:49.558857 231 external_mini_cluster.cc:1172] Attempting to check leaks for /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver pid 2993 I0507 06:28:49.559885 3109 generic_service.cc:171] Checking for leaks (request via RPC) I0507 06:28:49.620261 231 external_mini_cluster.cc:1131] Killing /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver with pid 2993 I0507 06:28:49.631449 231 external_mini_cluster.cc:1172] Attempting to check leaks for /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-master pid 2677 I0507 06:28:49.632575 2733 generic_service.cc:171] Checking for leaks (request via RPC) I0507 06:28:49.719729 231 external_mini_cluster.cc:1131] Killing /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-master with pid 2677 [ OK ] RaftConsensusNonVoterITest.NonVoterReplicasDoNotVote (12357 ms) [ RUN ] RaftConsensusNonVoterITest.PromoteAndDemote I0507 06:28:49.744477 231 test_util.cc:212] Using random seed: 1417102903 I0507 06:28:49.746285 231 ts_itest-base.cc:112] Starting cluster with: I0507 06:28:49.746341 231 ts_itest-base.cc:113] -------------- I0507 06:28:49.746359 231 ts_itest-base.cc:114] 4 tablet servers I0507 06:28:49.746394 231 ts_itest-base.cc:115] 3 replicas per TS I0507 06:28:49.746423 231 ts_itest-base.cc:116] -------------- I0507 06:28:49.747799 231 external_mini_cluster.cc:933] Running /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-master /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-master --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromoteAndDemote.1557210500371863-231/raft_consensus-itest-cluster/master-0/wal --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromoteAndDemote.1557210500371863-231/raft_consensus-itest-cluster/master-0/data --block_manager=log --webserver_interface=localhost --ipki_ca_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.0.57.254:39451 --webserver_interface=127.0.57.254 --webserver_port=0 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromoteAndDemote.1557210500371863-231/raft_consensus-itest-cluster/master-0/logs --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromoteAndDemote.1557210500371863-231/raft_consensus-itest-cluster/master-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --catalog_manager_wait_for_new_tablets_to_elect_leader=false --catalog_manager_evict_excess_replicas=false --rpc_reuseport=true with env {} WARNING: Logging before InitGoogleLogging() is written to STDERR W0507 06:28:49.953531 3317 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0507 06:28:49.953812 3317 flags.cc:404] Enabled unsafe flag: --never_fsync=true W0507 06:28:49.956415 3317 flags.cc:404] Enabled experimental flag: --ipki_ca_key_size=1024 W0507 06:28:49.956465 3317 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W0507 06:28:49.956507 3317 flags.cc:404] Enabled experimental flag: --tsk_num_rsa_bits=512 W0507 06:28:49.956568 3317 flags.cc:404] Enabled experimental flag: --rpc_reuseport=true I0507 06:28:49.961567 3317 master_main.cc:73] Master server non-default flags: --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromoteAndDemote.1557210500371863-231/raft_consensus-itest-cluster/master-0/data --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromoteAndDemote.1557210500371863-231/raft_consensus-itest-cluster/master-0/wal --catalog_manager_evict_excess_replicas=false --catalog_manager_wait_for_new_tablets_to_elect_leader=false --ipki_ca_key_size=1024 --ipki_server_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.0.57.254:39451 --rpc_reuseport=true --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromoteAndDemote.1557210500371863-231/raft_consensus-itest-cluster/master-0/data/info.pb --webserver_interface=127.0.57.254 --webserver_port=0 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromoteAndDemote.1557210500371863-231/raft_consensus-itest-cluster/master-0/logs --logbuflevel=-1 --logtostderr=true Master server version: kudu 1.10.0-cdh6.x-SNAPSHOT revision 7d0ee26697926e87af9ef395008799577f6ab43f build type FASTDEBUG built by jenkins at 06 May 2019 22:44:21 PST on quasar-knmpfx-1.vpc.cloudera.com build id 322017 ASAN enabled I0507 06:28:49.968483 3317 system_ntp.cc:202] NTP initialized. Skew: 500ppm Current error: 39016us I0507 06:28:49.969864 3317 fs_manager.cc:263] Metadata directory not provided I0507 06:28:49.969951 3317 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0507 06:28:49.970295 3317 server_base.cc:433] Could not load existing FS layout: Not found: could not find a healthy instance file I0507 06:28:49.970381 3317 server_base.cc:434] Attempting to create new FS layout instead I0507 06:28:49.972980 3317 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromoteAndDemote.1557210500371863-231/raft_consensus-itest-cluster/master-0/data/instance: uuid: "7c82905e75974072bf33c7f4986bbc1a" format_stamp: "Formatted at 2019-05-07 06:28:49 on dist-test-slave-dist-test-slave-rqp6" I0507 06:28:49.973584 3317 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromoteAndDemote.1557210500371863-231/raft_consensus-itest-cluster/master-0/wal/instance: uuid: "7c82905e75974072bf33c7f4986bbc1a" format_stamp: "Formatted at 2019-05-07 06:28:49 on dist-test-slave-dist-test-slave-rqp6" I0507 06:28:49.980432 3317 fs_manager.cc:503] Time spent creating directory manager: real 0.007s user 0.005s sys 0.001s I0507 06:28:49.981577 3317 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0507 06:28:49.982175 3317 file_cache.cc:466] Constructed file cache lbm with capacity 419430 I0507 06:28:49.985972 3317 fs_manager.cc:419] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s I0507 06:28:49.986058 3317 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromoteAndDemote.1557210500371863-231/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromoteAndDemote.1557210500371863-231/raft_consensus-itest-cluster/master-0/wal uuid: "7c82905e75974072bf33c7f4986bbc1a" format_stamp: "Formatted at 2019-05-07 06:28:49 on dist-test-slave-dist-test-slave-rqp6" I0507 06:28:49.986227 3317 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromoteAndDemote.1557210500371863-231/raft_consensus-itest-cluster/master-0/wal metadata directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromoteAndDemote.1557210500371863-231/raft_consensus-itest-cluster/master-0/wal 1 data directories: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromoteAndDemote.1557210500371863-231/raft_consensus-itest-cluster/master-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0507 06:28:50.014881 3317 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0507 06:28:50.030174 3317 rpc_server.cc:205] RPC server started. Bound to: 127.0.57.254:39451 I0507 06:28:50.031544 3317 webserver.cc:308] Webserver started at http://127.0.57.254:36619/ using document root and password file I0507 06:28:50.032450 3317 server_base.cc:608] Dumped server information to /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromoteAndDemote.1557210500371863-231/raft_consensus-itest-cluster/master-0/data/info.pb I0507 06:28:50.035359 231 external_mini_cluster.cc:995] Started /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-master as pid 3317 I0507 06:28:50.036128 231 external_mini_cluster.cc:933] Running /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromoteAndDemote.1557210500371863-231/raft_consensus-itest-cluster/ts-0/wal --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromoteAndDemote.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data --block_manager=log --rpc_bind_addresses=127.0.57.193:0 --local_ip_for_outbound_sockets=127.0.57.193 --webserver_interface=127.0.57.193 --webserver_port=0 --tserver_master_addrs=127.0.57.254:39451 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromoteAndDemote.1557210500371863-231/raft_consensus-itest-cluster/ts-0/logs --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromoteAndDemote.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_heartbeat_interval_ms=50 --evict_failed_followers=false --enable_leader_failure_detection=false with env {} I0507 06:28:50.036507 3376 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0507 06:28:50.049439 3376 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 7c82905e75974072bf33c7f4986bbc1a: Bootstrap starting. I0507 06:28:50.052345 3376 tablet_bootstrap.cc:590] T 00000000000000000000000000000000 P 7c82905e75974072bf33c7f4986bbc1a: No blocks or log segments found. Creating new log. I0507 06:28:50.053352 3376 log.cc:525] T 00000000000000000000000000000000 P 7c82905e75974072bf33c7f4986bbc1a: Log is configured to *not* fsync() on all Append() calls I0507 06:28:50.055411 3376 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 7c82905e75974072bf33c7f4986bbc1a: No bootstrap required, opened a new log I0507 06:28:50.062222 3376 raft_consensus.cc:340] T 00000000000000000000000000000000 P 7c82905e75974072bf33c7f4986bbc1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c82905e75974072bf33c7f4986bbc1a" member_type: VOTER } I0507 06:28:50.062477 3376 raft_consensus.cc:366] T 00000000000000000000000000000000 P 7c82905e75974072bf33c7f4986bbc1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0507 06:28:50.062543 3376 raft_consensus.cc:705] T 00000000000000000000000000000000 P 7c82905e75974072bf33c7f4986bbc1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c82905e75974072bf33c7f4986bbc1a, State: Initialized, Role: FOLLOWER I0507 06:28:50.063158 3376 consensus_queue.cc:229] T 00000000000000000000000000000000 P 7c82905e75974072bf33c7f4986bbc1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c82905e75974072bf33c7f4986bbc1a" member_type: VOTER } I0507 06:28:50.063360 3376 raft_consensus.cc:380] T 00000000000000000000000000000000 P 7c82905e75974072bf33c7f4986bbc1a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0507 06:28:50.063441 3376 raft_consensus.cc:472] T 00000000000000000000000000000000 P 7c82905e75974072bf33c7f4986bbc1a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0507 06:28:50.063550 3376 raft_consensus.cc:2886] T 00000000000000000000000000000000 P 7c82905e75974072bf33c7f4986bbc1a [term 0 FOLLOWER]: Advancing to term 1 WARNING: Logging before InitGoogleLogging() is written to STDERR W0507 06:28:50.206357 3377 flags.cc:404] Enabled unsafe flag: --enable_leader_failure_detection=false W0507 06:28:50.206619 3377 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0507 06:28:50.206691 3377 flags.cc:404] Enabled unsafe flag: --never_fsync=true W0507 06:28:50.208814 3377 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W0507 06:28:50.208930 3377 flags.cc:404] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.57.193 I0507 06:28:50.213440 3377 tablet_server_main.cc:78] Tablet server non-default flags: --enable_leader_failure_detection=false --evict_failed_followers=false --raft_heartbeat_interval_ms=50 --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromoteAndDemote.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromoteAndDemote.1557210500371863-231/raft_consensus-itest-cluster/ts-0/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.0.57.193:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromoteAndDemote.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data/info.pb --webserver_interface=127.0.57.193 --webserver_port=0 --tserver_master_addrs=127.0.57.254:39451 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.0.57.193 --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromoteAndDemote.1557210500371863-231/raft_consensus-itest-cluster/ts-0/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.10.0-cdh6.x-SNAPSHOT revision 7d0ee26697926e87af9ef395008799577f6ab43f build type FASTDEBUG built by jenkins at 06 May 2019 22:44:21 PST on quasar-knmpfx-1.vpc.cloudera.com build id 322017 ASAN enabled I0507 06:28:50.221156 3377 system_ntp.cc:202] NTP initialized. Skew: 500ppm Current error: 39516us I0507 06:28:50.221798 3377 fs_manager.cc:263] Metadata directory not provided I0507 06:28:50.221861 3377 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0507 06:28:50.222124 3377 server_base.cc:433] Could not load existing FS layout: Not found: could not find a healthy instance file I0507 06:28:50.222193 3377 server_base.cc:434] Attempting to create new FS layout instead I0507 06:28:50.224562 3377 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromoteAndDemote.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data/instance: uuid: "170627b774264bb6ab30a5080638e05f" format_stamp: "Formatted at 2019-05-07 06:28:50 on dist-test-slave-dist-test-slave-rqp6" I0507 06:28:50.225123 3377 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromoteAndDemote.1557210500371863-231/raft_consensus-itest-cluster/ts-0/wal/instance: uuid: "170627b774264bb6ab30a5080638e05f" format_stamp: "Formatted at 2019-05-07 06:28:50 on dist-test-slave-dist-test-slave-rqp6" I0507 06:28:50.231575 3377 fs_manager.cc:503] Time spent creating directory manager: real 0.006s user 0.001s sys 0.004s I0507 06:28:50.232622 3377 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0507 06:28:50.233203 3377 file_cache.cc:466] Constructed file cache lbm with capacity 419430 I0507 06:28:50.236183 3377 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.000s sys 0.000s I0507 06:28:50.236265 3377 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromoteAndDemote.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromoteAndDemote.1557210500371863-231/raft_consensus-itest-cluster/ts-0/wal uuid: "170627b774264bb6ab30a5080638e05f" format_stamp: "Formatted at 2019-05-07 06:28:50 on dist-test-slave-dist-test-slave-rqp6" I0507 06:28:50.236404 3377 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromoteAndDemote.1557210500371863-231/raft_consensus-itest-cluster/ts-0/wal metadata directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromoteAndDemote.1557210500371863-231/raft_consensus-itest-cluster/ts-0/wal 1 data directories: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromoteAndDemote.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0507 06:28:50.286799 3377 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0507 06:28:50.288506 3377 ts_tablet_manager.cc:356] Loaded tablet metadata (0 total tablets, 0 live tablets) I0507 06:28:50.288612 3377 ts_tablet_manager.cc:376] Registered 0 tablets I0507 06:28:50.310550 3377 rpc_server.cc:205] RPC server started. Bound to: 127.0.57.193:46551 I0507 06:28:50.311651 3377 webserver.cc:308] Webserver started at http://127.0.57.193:39701/ using document root and password file I0507 06:28:50.312311 3377 server_base.cc:608] Dumped server information to /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromoteAndDemote.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data/info.pb I0507 06:28:50.317454 231 external_mini_cluster.cc:995] Started /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver as pid 3377 I0507 06:28:50.318111 231 external_mini_cluster.cc:933] Running /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromoteAndDemote.1557210500371863-231/raft_consensus-itest-cluster/ts-1/wal --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromoteAndDemote.1557210500371863-231/raft_consensus-itest-cluster/ts-1/data --block_manager=log --rpc_bind_addresses=127.0.57.194:0 --local_ip_for_outbound_sockets=127.0.57.194 --webserver_interface=127.0.57.194 --webserver_port=0 --tserver_master_addrs=127.0.57.254:39451 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromoteAndDemote.1557210500371863-231/raft_consensus-itest-cluster/ts-1/logs --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromoteAndDemote.1557210500371863-231/raft_consensus-itest-cluster/ts-1/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_heartbeat_interval_ms=50 --evict_failed_followers=false --enable_leader_failure_detection=false with env {} I0507 06:28:50.334081 3497 heartbeater.cc:345] Connected to a master server at 127.0.57.254:39451 I0507 06:28:50.334306 3497 heartbeater.cc:426] Registering TS with master... I0507 06:28:50.334870 3497 heartbeater.cc:475] Master 127.0.57.254:39451 requested a full tablet report, sending... W0507 06:28:50.337457 3497 heartbeater.cc:587] Failed to heartbeat to 127.0.57.254:39451 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting I0507 06:28:50.337636 3497 heartbeater.cc:426] Registering TS with master... I0507 06:28:50.337868 3497 heartbeater.cc:475] Master 127.0.57.254:39451 requested a full tablet report, sending... I0507 06:28:50.338990 3497 heartbeater.cc:426] Registering TS with master... I0507 06:28:50.339227 3497 heartbeater.cc:475] Master 127.0.57.254:39451 requested a full tablet report, sending... W0507 06:28:50.340159 3497 heartbeater.cc:380] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts. WARNING: Logging before InitGoogleLogging() is written to STDERR W0507 06:28:50.478093 3501 flags.cc:404] Enabled unsafe flag: --enable_leader_failure_detection=false W0507 06:28:50.478351 3501 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0507 06:28:50.478408 3501 flags.cc:404] Enabled unsafe flag: --never_fsync=true W0507 06:28:50.480481 3501 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W0507 06:28:50.480686 3501 flags.cc:404] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.57.194 I0507 06:28:50.484921 3501 tablet_server_main.cc:78] Tablet server non-default flags: --enable_leader_failure_detection=false --evict_failed_followers=false --raft_heartbeat_interval_ms=50 --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromoteAndDemote.1557210500371863-231/raft_consensus-itest-cluster/ts-1/data --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromoteAndDemote.1557210500371863-231/raft_consensus-itest-cluster/ts-1/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.0.57.194:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromoteAndDemote.1557210500371863-231/raft_consensus-itest-cluster/ts-1/data/info.pb --webserver_interface=127.0.57.194 --webserver_port=0 --tserver_master_addrs=127.0.57.254:39451 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.0.57.194 --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromoteAndDemote.1557210500371863-231/raft_consensus-itest-cluster/ts-1/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.10.0-cdh6.x-SNAPSHOT revision 7d0ee26697926e87af9ef395008799577f6ab43f build type FASTDEBUG built by jenkins at 06 May 2019 22:44:21 PST on quasar-knmpfx-1.vpc.cloudera.com build id 322017 ASAN enabled I0507 06:28:50.493080 3501 system_ntp.cc:202] NTP initialized. Skew: 500ppm Current error: 39516us I0507 06:28:50.493788 3501 fs_manager.cc:263] Metadata directory not provided I0507 06:28:50.493851 3501 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0507 06:28:50.494114 3501 server_base.cc:433] Could not load existing FS layout: Not found: could not find a healthy instance file I0507 06:28:50.494182 3501 server_base.cc:434] Attempting to create new FS layout instead I0507 06:28:50.496635 3501 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromoteAndDemote.1557210500371863-231/raft_consensus-itest-cluster/ts-1/data/instance: uuid: "c8068fde2be746779966e9a1037d779c" format_stamp: "Formatted at 2019-05-07 06:28:50 on dist-test-slave-dist-test-slave-rqp6" I0507 06:28:50.497189 3501 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromoteAndDemote.1557210500371863-231/raft_consensus-itest-cluster/ts-1/wal/instance: uuid: "c8068fde2be746779966e9a1037d779c" format_stamp: "Formatted at 2019-05-07 06:28:50 on dist-test-slave-dist-test-slave-rqp6" I0507 06:28:50.503597 3501 fs_manager.cc:503] Time spent creating directory manager: real 0.006s user 0.004s sys 0.001s I0507 06:28:50.504856 3501 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0507 06:28:50.505416 3501 file_cache.cc:466] Constructed file cache lbm with capacity 419430 I0507 06:28:50.508280 3501 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s I0507 06:28:50.508368 3501 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromoteAndDemote.1557210500371863-231/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromoteAndDemote.1557210500371863-231/raft_consensus-itest-cluster/ts-1/wal uuid: "c8068fde2be746779966e9a1037d779c" format_stamp: "Formatted at 2019-05-07 06:28:50 on dist-test-slave-dist-test-slave-rqp6" I0507 06:28:50.508517 3501 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromoteAndDemote.1557210500371863-231/raft_consensus-itest-cluster/ts-1/wal metadata directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromoteAndDemote.1557210500371863-231/raft_consensus-itest-cluster/ts-1/wal 1 data directories: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromoteAndDemote.1557210500371863-231/raft_consensus-itest-cluster/ts-1/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0507 06:28:50.627677 3501 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0507 06:28:50.629402 3501 ts_tablet_manager.cc:356] Loaded tablet metadata (0 total tablets, 0 live tablets) I0507 06:28:50.629485 3501 ts_tablet_manager.cc:376] Registered 0 tablets I0507 06:28:50.653239 3501 rpc_server.cc:205] RPC server started. Bound to: 127.0.57.194:40627 I0507 06:28:50.654371 3501 webserver.cc:308] Webserver started at http://127.0.57.194:39775/ using document root and password file I0507 06:28:50.655059 3501 server_base.cc:608] Dumped server information to /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromoteAndDemote.1557210500371863-231/raft_consensus-itest-cluster/ts-1/data/info.pb I0507 06:28:50.657239 231 external_mini_cluster.cc:995] Started /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver as pid 3501 I0507 06:28:50.657886 231 external_mini_cluster.cc:933] Running /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromoteAndDemote.1557210500371863-231/raft_consensus-itest-cluster/ts-2/wal --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromoteAndDemote.1557210500371863-231/raft_consensus-itest-cluster/ts-2/data --block_manager=log --rpc_bind_addresses=127.0.57.195:0 --local_ip_for_outbound_sockets=127.0.57.195 --webserver_interface=127.0.57.195 --webserver_port=0 --tserver_master_addrs=127.0.57.254:39451 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromoteAndDemote.1557210500371863-231/raft_consensus-itest-cluster/ts-2/logs --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromoteAndDemote.1557210500371863-231/raft_consensus-itest-cluster/ts-2/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_heartbeat_interval_ms=50 --evict_failed_followers=false --enable_leader_failure_detection=false with env {} I0507 06:28:50.668388 3620 heartbeater.cc:345] Connected to a master server at 127.0.57.254:39451 I0507 06:28:50.668642 3620 heartbeater.cc:426] Registering TS with master... I0507 06:28:50.669178 3620 heartbeater.cc:475] Master 127.0.57.254:39451 requested a full tablet report, sending... W0507 06:28:50.670804 3620 heartbeater.cc:587] Failed to heartbeat to 127.0.57.254:39451 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting I0507 06:28:50.670934 3620 heartbeater.cc:426] Registering TS with master... I0507 06:28:50.671131 3620 heartbeater.cc:475] Master 127.0.57.254:39451 requested a full tablet report, sending... I0507 06:28:50.672130 3620 heartbeater.cc:426] Registering TS with master... I0507 06:28:50.672283 3620 heartbeater.cc:475] Master 127.0.57.254:39451 requested a full tablet report, sending... W0507 06:28:50.673159 3620 heartbeater.cc:380] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts. W0507 06:28:50.790624 3376 consensus_meta.cc:220] T 00000000000000000000000000000000 P 7c82905e75974072bf33c7f4986bbc1a: Time spent flushing consensus metadata: real 0.727s user 0.005s sys 0.000s I0507 06:28:50.790769 3376 raft_consensus.cc:494] T 00000000000000000000000000000000 P 7c82905e75974072bf33c7f4986bbc1a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c82905e75974072bf33c7f4986bbc1a" member_type: VOTER } I0507 06:28:50.791589 3376 leader_election.cc:310] T 00000000000000000000000000000000 P 7c82905e75974072bf33c7f4986bbc1a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7c82905e75974072bf33c7f4986bbc1a; no voters: I0507 06:28:50.792721 3376 leader_election.cc:296] T 00000000000000000000000000000000 P 7c82905e75974072bf33c7f4986bbc1a [CANDIDATE]: Term 1 election: Requested vote from peers I0507 06:28:50.793025 3624 raft_consensus.cc:2640] T 00000000000000000000000000000000 P 7c82905e75974072bf33c7f4986bbc1a [term 1 FOLLOWER]: Leader election won for term 1 I0507 06:28:50.794134 3624 raft_consensus.cc:667] T 00000000000000000000000000000000 P 7c82905e75974072bf33c7f4986bbc1a [term 1 LEADER]: Becoming Leader. State: Replica: 7c82905e75974072bf33c7f4986bbc1a, State: Running, Role: LEADER I0507 06:28:50.794652 3624 consensus_queue.cc:206] T 00000000000000000000000000000000 P 7c82905e75974072bf33c7f4986bbc1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c82905e75974072bf33c7f4986bbc1a" member_type: VOTER } I0507 06:28:50.797263 3376 sys_catalog.cc:337] T 00000000000000000000000000000000 P 7c82905e75974072bf33c7f4986bbc1a [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 1 leader_uuid: "7c82905e75974072bf33c7f4986bbc1a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c82905e75974072bf33c7f4986bbc1a" member_type: VOTER } } I0507 06:28:50.797158 3625 sys_catalog.cc:337] T 00000000000000000000000000000000 P 7c82905e75974072bf33c7f4986bbc1a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7c82905e75974072bf33c7f4986bbc1a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c82905e75974072bf33c7f4986bbc1a" member_type: VOTER } } I0507 06:28:50.800801 3625 sys_catalog.cc:340] T 00000000000000000000000000000000 P 7c82905e75974072bf33c7f4986bbc1a [sys.catalog]: This master's current role is: LEADER I0507 06:28:50.801187 3376 sys_catalog.cc:340] T 00000000000000000000000000000000 P 7c82905e75974072bf33c7f4986bbc1a [sys.catalog]: This master's current role is: LEADER I0507 06:28:50.800379 3626 sys_catalog.cc:337] T 00000000000000000000000000000000 P 7c82905e75974072bf33c7f4986bbc1a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7c82905e75974072bf33c7f4986bbc1a. Latest consensus state: current_term: 1 leader_uuid: "7c82905e75974072bf33c7f4986bbc1a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c82905e75974072bf33c7f4986bbc1a" member_type: VOTER } } I0507 06:28:50.803105 3376 sys_catalog.cc:424] T 00000000000000000000000000000000 P 7c82905e75974072bf33c7f4986bbc1a [sys.catalog]: configured and running, proceeding with master startup. I0507 06:28:50.803310 3626 sys_catalog.cc:340] T 00000000000000000000000000000000 P 7c82905e75974072bf33c7f4986bbc1a [sys.catalog]: This master's current role is: LEADER I0507 06:28:50.803400 3628 catalog_manager.cc:1055] Loading table and tablet metadata into memory... I0507 06:28:50.815316 3628 catalog_manager.cc:1066] Initializing Kudu internal certificate authority... WARNING: Logging before InitGoogleLogging() is written to STDERR W0507 06:28:50.815943 3623 flags.cc:404] Enabled unsafe flag: --enable_leader_failure_detection=false W0507 06:28:50.816207 3623 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0507 06:28:50.816283 3623 flags.cc:404] Enabled unsafe flag: --never_fsync=true W0507 06:28:50.818536 3623 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W0507 06:28:50.818645 3623 flags.cc:404] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.57.195 I0507 06:28:50.832159 3623 tablet_server_main.cc:78] Tablet server non-default flags: --enable_leader_failure_detection=false --evict_failed_followers=false --raft_heartbeat_interval_ms=50 --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromoteAndDemote.1557210500371863-231/raft_consensus-itest-cluster/ts-2/data --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromoteAndDemote.1557210500371863-231/raft_consensus-itest-cluster/ts-2/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.0.57.195:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromoteAndDemote.1557210500371863-231/raft_consensus-itest-cluster/ts-2/data/info.pb --webserver_interface=127.0.57.195 --webserver_port=0 --tserver_master_addrs=127.0.57.254:39451 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.0.57.195 --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromoteAndDemote.1557210500371863-231/raft_consensus-itest-cluster/ts-2/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.10.0-cdh6.x-SNAPSHOT revision 7d0ee26697926e87af9ef395008799577f6ab43f build type FASTDEBUG built by jenkins at 06 May 2019 22:44:21 PST on quasar-knmpfx-1.vpc.cloudera.com build id 322017 ASAN enabled I0507 06:28:50.839788 3623 system_ntp.cc:202] NTP initialized. Skew: 500ppm Current error: 39516us I0507 06:28:50.840477 3623 fs_manager.cc:263] Metadata directory not provided I0507 06:28:50.840553 3623 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0507 06:28:50.840816 3623 server_base.cc:433] Could not load existing FS layout: Not found: could not find a healthy instance file I0507 06:28:50.840879 3623 server_base.cc:434] Attempting to create new FS layout instead I0507 06:28:50.843452 3623 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromoteAndDemote.1557210500371863-231/raft_consensus-itest-cluster/ts-2/data/instance: uuid: "5855917eaa4d4a4c96f9c0e36124ea7c" format_stamp: "Formatted at 2019-05-07 06:28:50 on dist-test-slave-dist-test-slave-rqp6" I0507 06:28:50.843991 3623 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromoteAndDemote.1557210500371863-231/raft_consensus-itest-cluster/ts-2/wal/instance: uuid: "5855917eaa4d4a4c96f9c0e36124ea7c" format_stamp: "Formatted at 2019-05-07 06:28:50 on dist-test-slave-dist-test-slave-rqp6" I0507 06:28:50.850347 3623 fs_manager.cc:503] Time spent creating directory manager: real 0.006s user 0.006s sys 0.002s I0507 06:28:50.851388 3623 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0507 06:28:50.851931 3623 file_cache.cc:466] Constructed file cache lbm with capacity 419430 I0507 06:28:50.854909 3623 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s I0507 06:28:50.855029 3623 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromoteAndDemote.1557210500371863-231/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromoteAndDemote.1557210500371863-231/raft_consensus-itest-cluster/ts-2/wal uuid: "5855917eaa4d4a4c96f9c0e36124ea7c" format_stamp: "Formatted at 2019-05-07 06:28:50 on dist-test-slave-dist-test-slave-rqp6" I0507 06:28:50.855167 3623 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromoteAndDemote.1557210500371863-231/raft_consensus-itest-cluster/ts-2/wal metadata directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromoteAndDemote.1557210500371863-231/raft_consensus-itest-cluster/ts-2/wal 1 data directories: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromoteAndDemote.1557210500371863-231/raft_consensus-itest-cluster/ts-2/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0507 06:28:50.862578 3628 catalog_manager.cc:950] Generated new certificate authority record I0507 06:28:50.863907 3628 catalog_manager.cc:1075] Loading token signing keys... I0507 06:28:50.870285 3628 catalog_manager.cc:4292] T 00000000000000000000000000000000 P 7c82905e75974072bf33c7f4986bbc1a: Generated new TSK 0 I0507 06:28:50.898785 3623 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0507 06:28:50.900918 3623 ts_tablet_manager.cc:356] Loaded tablet metadata (0 total tablets, 0 live tablets) I0507 06:28:50.901016 3623 ts_tablet_manager.cc:376] Registered 0 tablets I0507 06:28:50.925933 3623 rpc_server.cc:205] RPC server started. Bound to: 127.0.57.195:41035 I0507 06:28:50.927052 3623 webserver.cc:308] Webserver started at http://127.0.57.195:35537/ using document root and password file I0507 06:28:50.927673 3623 server_base.cc:608] Dumped server information to /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromoteAndDemote.1557210500371863-231/raft_consensus-itest-cluster/ts-2/data/info.pb I0507 06:28:50.935264 231 external_mini_cluster.cc:995] Started /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver as pid 3623 I0507 06:28:50.936014 231 external_mini_cluster.cc:933] Running /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromoteAndDemote.1557210500371863-231/raft_consensus-itest-cluster/ts-3/wal --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromoteAndDemote.1557210500371863-231/raft_consensus-itest-cluster/ts-3/data --block_manager=log --rpc_bind_addresses=127.0.57.196:0 --local_ip_for_outbound_sockets=127.0.57.196 --webserver_interface=127.0.57.196 --webserver_port=0 --tserver_master_addrs=127.0.57.254:39451 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromoteAndDemote.1557210500371863-231/raft_consensus-itest-cluster/ts-3/logs --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromoteAndDemote.1557210500371863-231/raft_consensus-itest-cluster/ts-3/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_heartbeat_interval_ms=50 --evict_failed_followers=false --enable_leader_failure_detection=false with env {} I0507 06:28:50.946836 3751 heartbeater.cc:345] Connected to a master server at 127.0.57.254:39451 I0507 06:28:50.947072 3751 heartbeater.cc:426] Registering TS with master... I0507 06:28:50.947644 3751 heartbeater.cc:475] Master 127.0.57.254:39451 requested a full tablet report, sending... I0507 06:28:50.949092 3343 ts_manager.cc:159] Registered new tserver with Master: 5855917eaa4d4a4c96f9c0e36124ea7c (127.0.57.195:41035) I0507 06:28:50.950886 3343 master_service.cc:273] Signed X509 certificate for tserver {username='slave'} at 127.0.57.195:44251 WARNING: Logging before InitGoogleLogging() is written to STDERR W0507 06:28:51.105697 3754 flags.cc:404] Enabled unsafe flag: --enable_leader_failure_detection=false W0507 06:28:51.106073 3754 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0507 06:28:51.106148 3754 flags.cc:404] Enabled unsafe flag: --never_fsync=true W0507 06:28:51.108351 3754 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W0507 06:28:51.108474 3754 flags.cc:404] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.57.196 I0507 06:28:51.116737 3754 tablet_server_main.cc:78] Tablet server non-default flags: --enable_leader_failure_detection=false --evict_failed_followers=false --raft_heartbeat_interval_ms=50 --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromoteAndDemote.1557210500371863-231/raft_consensus-itest-cluster/ts-3/data --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromoteAndDemote.1557210500371863-231/raft_consensus-itest-cluster/ts-3/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.0.57.196:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromoteAndDemote.1557210500371863-231/raft_consensus-itest-cluster/ts-3/data/info.pb --webserver_interface=127.0.57.196 --webserver_port=0 --tserver_master_addrs=127.0.57.254:39451 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.0.57.196 --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromoteAndDemote.1557210500371863-231/raft_consensus-itest-cluster/ts-3/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.10.0-cdh6.x-SNAPSHOT revision 7d0ee26697926e87af9ef395008799577f6ab43f build type FASTDEBUG built by jenkins at 06 May 2019 22:44:21 PST on quasar-knmpfx-1.vpc.cloudera.com build id 322017 ASAN enabled I0507 06:28:51.125864 3754 system_ntp.cc:202] NTP initialized. Skew: 500ppm Current error: 40016us I0507 06:28:51.126561 3754 fs_manager.cc:263] Metadata directory not provided I0507 06:28:51.126629 3754 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0507 06:28:51.127010 3754 server_base.cc:433] Could not load existing FS layout: Not found: could not find a healthy instance file I0507 06:28:51.127077 3754 server_base.cc:434] Attempting to create new FS layout instead I0507 06:28:51.129799 3754 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromoteAndDemote.1557210500371863-231/raft_consensus-itest-cluster/ts-3/data/instance: uuid: "eeb9cd29752c4282b6955954cf29243a" format_stamp: "Formatted at 2019-05-07 06:28:51 on dist-test-slave-dist-test-slave-rqp6" I0507 06:28:51.130393 3754 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromoteAndDemote.1557210500371863-231/raft_consensus-itest-cluster/ts-3/wal/instance: uuid: "eeb9cd29752c4282b6955954cf29243a" format_stamp: "Formatted at 2019-05-07 06:28:51 on dist-test-slave-dist-test-slave-rqp6" I0507 06:28:51.137689 3754 fs_manager.cc:503] Time spent creating directory manager: real 0.007s user 0.002s sys 0.005s I0507 06:28:51.138876 3754 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0507 06:28:51.139564 3754 file_cache.cc:466] Constructed file cache lbm with capacity 419430 I0507 06:28:51.142668 3754 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s I0507 06:28:51.142756 3754 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromoteAndDemote.1557210500371863-231/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromoteAndDemote.1557210500371863-231/raft_consensus-itest-cluster/ts-3/wal uuid: "eeb9cd29752c4282b6955954cf29243a" format_stamp: "Formatted at 2019-05-07 06:28:51 on dist-test-slave-dist-test-slave-rqp6" I0507 06:28:51.142951 3754 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromoteAndDemote.1557210500371863-231/raft_consensus-itest-cluster/ts-3/wal metadata directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromoteAndDemote.1557210500371863-231/raft_consensus-itest-cluster/ts-3/wal 1 data directories: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromoteAndDemote.1557210500371863-231/raft_consensus-itest-cluster/ts-3/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0507 06:28:51.186189 3754 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0507 06:28:51.187918 3754 ts_tablet_manager.cc:356] Loaded tablet metadata (0 total tablets, 0 live tablets) I0507 06:28:51.188014 3754 ts_tablet_manager.cc:376] Registered 0 tablets I0507 06:28:51.216979 3754 rpc_server.cc:205] RPC server started. Bound to: 127.0.57.196:40737 I0507 06:28:51.218174 3754 webserver.cc:308] Webserver started at http://127.0.57.196:41011/ using document root and password file I0507 06:28:51.218849 3754 server_base.cc:608] Dumped server information to /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromoteAndDemote.1557210500371863-231/raft_consensus-itest-cluster/ts-3/data/info.pb I0507 06:28:51.223618 231 external_mini_cluster.cc:995] Started /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver as pid 3754 I0507 06:28:51.234206 3873 heartbeater.cc:345] Connected to a master server at 127.0.57.254:39451 I0507 06:28:51.234478 3873 heartbeater.cc:426] Registering TS with master... I0507 06:28:51.235107 3873 heartbeater.cc:475] Master 127.0.57.254:39451 requested a full tablet report, sending... I0507 06:28:51.236393 3343 ts_manager.cc:159] Registered new tserver with Master: eeb9cd29752c4282b6955954cf29243a (127.0.57.196:40737) I0507 06:28:51.237509 3343 master_service.cc:273] Signed X509 certificate for tserver {username='slave'} at 127.0.57.196:38815 I0507 06:28:51.340382 3497 heartbeater.cc:426] Registering TS with master... I0507 06:28:51.340819 3497 heartbeater.cc:475] Master 127.0.57.254:39451 requested a full tablet report, sending... I0507 06:28:51.341842 3343 ts_manager.cc:159] Registered new tserver with Master: 170627b774264bb6ab30a5080638e05f (127.0.57.193:46551) I0507 06:28:51.342882 3343 master_service.cc:273] Signed X509 certificate for tserver {username='slave'} at 127.0.57.193:35901 I0507 06:28:51.673339 3620 heartbeater.cc:426] Registering TS with master... I0507 06:28:51.673733 3620 heartbeater.cc:475] Master 127.0.57.254:39451 requested a full tablet report, sending... I0507 06:28:51.674681 3343 ts_manager.cc:159] Registered new tserver with Master: c8068fde2be746779966e9a1037d779c (127.0.57.194:40627) I0507 06:28:51.675653 3343 master_service.cc:273] Signed X509 certificate for tserver {username='slave'} at 127.0.57.194:53071 I0507 06:28:51.676404 231 external_mini_cluster.cc:623] 4 TS(s) registered with all masters I0507 06:28:51.687577 3342 catalog_manager.cc:1415] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38798: name: "TestTable" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 3 split_rows_range_bounds { } partition_schema { range_schema { columns { name: "key" } } } I0507 06:28:51.718421 3810 tablet_service.cc:1016] Processing CreateTablet for tablet dd440bce55914e85a2058ea4a1592ef0 (table=TestTable [id=736ce4d3f4d8476bacf42dd4da17bced]), partition=RANGE (key) PARTITION UNBOUNDED I0507 06:28:51.719322 3810 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet dd440bce55914e85a2058ea4a1592ef0. 1 dirs total, 0 dirs full, 0 dirs failed I0507 06:28:51.719436 3688 tablet_service.cc:1016] Processing CreateTablet for tablet dd440bce55914e85a2058ea4a1592ef0 (table=TestTable [id=736ce4d3f4d8476bacf42dd4da17bced]), partition=RANGE (key) PARTITION UNBOUNDED I0507 06:28:51.720221 3688 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet dd440bce55914e85a2058ea4a1592ef0. 1 dirs total, 0 dirs full, 0 dirs failed I0507 06:28:51.720337 3557 tablet_service.cc:1016] Processing CreateTablet for tablet dd440bce55914e85a2058ea4a1592ef0 (table=TestTable [id=736ce4d3f4d8476bacf42dd4da17bced]), partition=RANGE (key) PARTITION UNBOUNDED I0507 06:28:51.721323 3557 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet dd440bce55914e85a2058ea4a1592ef0. 1 dirs total, 0 dirs full, 0 dirs failed I0507 06:28:51.729040 3892 tablet_bootstrap.cc:439] T dd440bce55914e85a2058ea4a1592ef0 P eeb9cd29752c4282b6955954cf29243a: Bootstrap starting. I0507 06:28:51.729692 3893 tablet_bootstrap.cc:439] T dd440bce55914e85a2058ea4a1592ef0 P 5855917eaa4d4a4c96f9c0e36124ea7c: Bootstrap starting. I0507 06:28:51.732038 3892 tablet_bootstrap.cc:590] T dd440bce55914e85a2058ea4a1592ef0 P eeb9cd29752c4282b6955954cf29243a: No blocks or log segments found. Creating new log. I0507 06:28:51.733178 3894 tablet_bootstrap.cc:439] T dd440bce55914e85a2058ea4a1592ef0 P c8068fde2be746779966e9a1037d779c: Bootstrap starting. I0507 06:28:51.735555 3894 tablet_bootstrap.cc:590] T dd440bce55914e85a2058ea4a1592ef0 P c8068fde2be746779966e9a1037d779c: No blocks or log segments found. Creating new log. I0507 06:28:51.736449 3894 log.cc:525] T dd440bce55914e85a2058ea4a1592ef0 P c8068fde2be746779966e9a1037d779c: Log is configured to *not* fsync() on all Append() calls I0507 06:28:51.738555 3893 tablet_bootstrap.cc:590] T dd440bce55914e85a2058ea4a1592ef0 P 5855917eaa4d4a4c96f9c0e36124ea7c: No blocks or log segments found. Creating new log. I0507 06:28:51.739389 3893 log.cc:525] T dd440bce55914e85a2058ea4a1592ef0 P 5855917eaa4d4a4c96f9c0e36124ea7c: Log is configured to *not* fsync() on all Append() calls I0507 06:28:51.740970 3892 log.cc:525] T dd440bce55914e85a2058ea4a1592ef0 P eeb9cd29752c4282b6955954cf29243a: Log is configured to *not* fsync() on all Append() calls I0507 06:28:51.742697 3892 tablet_bootstrap.cc:439] T dd440bce55914e85a2058ea4a1592ef0 P eeb9cd29752c4282b6955954cf29243a: No bootstrap required, opened a new log I0507 06:28:51.742699 3893 tablet_bootstrap.cc:439] T dd440bce55914e85a2058ea4a1592ef0 P 5855917eaa4d4a4c96f9c0e36124ea7c: No bootstrap required, opened a new log I0507 06:28:51.743005 3892 ts_tablet_manager.cc:1053] T dd440bce55914e85a2058ea4a1592ef0 P eeb9cd29752c4282b6955954cf29243a: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.001s I0507 06:28:51.742998 3893 ts_tablet_manager.cc:1053] T dd440bce55914e85a2058ea4a1592ef0 P 5855917eaa4d4a4c96f9c0e36124ea7c: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.000s I0507 06:28:51.743149 3894 tablet_bootstrap.cc:439] T dd440bce55914e85a2058ea4a1592ef0 P c8068fde2be746779966e9a1037d779c: No bootstrap required, opened a new log I0507 06:28:51.743451 3894 ts_tablet_manager.cc:1053] T dd440bce55914e85a2058ea4a1592ef0 P c8068fde2be746779966e9a1037d779c: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.000s I0507 06:28:51.748560 3892 raft_consensus.cc:340] T dd440bce55914e85a2058ea4a1592ef0 P eeb9cd29752c4282b6955954cf29243a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeb9cd29752c4282b6955954cf29243a" member_type: VOTER last_known_addr { host: "127.0.57.196" port: 40737 } } peers { permanent_uuid: "c8068fde2be746779966e9a1037d779c" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 40627 } } peers { permanent_uuid: "5855917eaa4d4a4c96f9c0e36124ea7c" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 41035 } } I0507 06:28:51.748926 3892 raft_consensus.cc:705] T dd440bce55914e85a2058ea4a1592ef0 P eeb9cd29752c4282b6955954cf29243a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eeb9cd29752c4282b6955954cf29243a, State: Initialized, Role: FOLLOWER I0507 06:28:51.749243 3892 consensus_queue.cc:229] T dd440bce55914e85a2058ea4a1592ef0 P eeb9cd29752c4282b6955954cf29243a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeb9cd29752c4282b6955954cf29243a" member_type: VOTER last_known_addr { host: "127.0.57.196" port: 40737 } } peers { permanent_uuid: "c8068fde2be746779966e9a1037d779c" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 40627 } } peers { permanent_uuid: "5855917eaa4d4a4c96f9c0e36124ea7c" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 41035 } } I0507 06:28:51.750351 3873 heartbeater.cc:467] Master 127.0.57.254:39451 was elected leader, sending a full tablet report... I0507 06:28:51.750705 3892 ts_tablet_manager.cc:1081] T dd440bce55914e85a2058ea4a1592ef0 P eeb9cd29752c4282b6955954cf29243a: Time spent starting tablet: real 0.008s user 0.007s sys 0.001s I0507 06:28:51.750685 3893 raft_consensus.cc:340] T dd440bce55914e85a2058ea4a1592ef0 P 5855917eaa4d4a4c96f9c0e36124ea7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeb9cd29752c4282b6955954cf29243a" member_type: VOTER last_known_addr { host: "127.0.57.196" port: 40737 } } peers { permanent_uuid: "c8068fde2be746779966e9a1037d779c" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 40627 } } peers { permanent_uuid: "5855917eaa4d4a4c96f9c0e36124ea7c" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 41035 } } I0507 06:28:51.751081 3893 raft_consensus.cc:705] T dd440bce55914e85a2058ea4a1592ef0 P 5855917eaa4d4a4c96f9c0e36124ea7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5855917eaa4d4a4c96f9c0e36124ea7c, State: Initialized, Role: FOLLOWER I0507 06:28:51.751453 3893 consensus_queue.cc:229] T dd440bce55914e85a2058ea4a1592ef0 P 5855917eaa4d4a4c96f9c0e36124ea7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeb9cd29752c4282b6955954cf29243a" member_type: VOTER last_known_addr { host: "127.0.57.196" port: 40737 } } peers { permanent_uuid: "c8068fde2be746779966e9a1037d779c" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 40627 } } peers { permanent_uuid: "5855917eaa4d4a4c96f9c0e36124ea7c" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 41035 } } I0507 06:28:51.752187 3894 raft_consensus.cc:340] T dd440bce55914e85a2058ea4a1592ef0 P c8068fde2be746779966e9a1037d779c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeb9cd29752c4282b6955954cf29243a" member_type: VOTER last_known_addr { host: "127.0.57.196" port: 40737 } } peers { permanent_uuid: "c8068fde2be746779966e9a1037d779c" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 40627 } } peers { permanent_uuid: "5855917eaa4d4a4c96f9c0e36124ea7c" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 41035 } } I0507 06:28:51.752820 3894 raft_consensus.cc:705] T dd440bce55914e85a2058ea4a1592ef0 P c8068fde2be746779966e9a1037d779c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8068fde2be746779966e9a1037d779c, State: Initialized, Role: FOLLOWER I0507 06:28:51.752980 3751 heartbeater.cc:467] Master 127.0.57.254:39451 was elected leader, sending a full tablet report... I0507 06:28:51.753172 3894 consensus_queue.cc:229] T dd440bce55914e85a2058ea4a1592ef0 P c8068fde2be746779966e9a1037d779c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeb9cd29752c4282b6955954cf29243a" member_type: VOTER last_known_addr { host: "127.0.57.196" port: 40737 } } peers { permanent_uuid: "c8068fde2be746779966e9a1037d779c" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 40627 } } peers { permanent_uuid: "5855917eaa4d4a4c96f9c0e36124ea7c" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 41035 } } I0507 06:28:51.753484 3893 ts_tablet_manager.cc:1081] T dd440bce55914e85a2058ea4a1592ef0 P 5855917eaa4d4a4c96f9c0e36124ea7c: Time spent starting tablet: real 0.010s user 0.009s sys 0.002s I0507 06:28:51.754076 3620 heartbeater.cc:467] Master 127.0.57.254:39451 was elected leader, sending a full tablet report... I0507 06:28:51.754628 3894 ts_tablet_manager.cc:1081] T dd440bce55914e85a2058ea4a1592ef0 P c8068fde2be746779966e9a1037d779c: Time spent starting tablet: real 0.011s user 0.006s sys 0.004s I0507 06:28:51.760784 231 external_mini_cluster.cc:623] 4 TS(s) registered with all masters I0507 06:28:51.762274 231 ts_itest-base.cc:232] Waiting for 0 tablets on tserver 170627b774264bb6ab30a5080638e05f to finish bootstrapping I0507 06:28:51.774994 231 ts_itest-base.cc:232] Waiting for 1 tablets on tserver c8068fde2be746779966e9a1037d779c to finish bootstrapping I0507 06:28:51.781718 231 ts_itest-base.cc:232] Waiting for 1 tablets on tserver 5855917eaa4d4a4c96f9c0e36124ea7c to finish bootstrapping I0507 06:28:51.788028 231 ts_itest-base.cc:232] Waiting for 1 tablets on tserver eeb9cd29752c4282b6955954cf29243a to finish bootstrapping I0507 06:28:51.800434 3708 tablet_service.cc:1403] Received Run Leader Election RPC: tablet_id: "dd440bce55914e85a2058ea4a1592ef0" dest_uuid: "5855917eaa4d4a4c96f9c0e36124ea7c" from {username='slave'} at 127.0.0.1:33608 I0507 06:28:51.800751 3708 raft_consensus.cc:472] T dd440bce55914e85a2058ea4a1592ef0 P 5855917eaa4d4a4c96f9c0e36124ea7c [term 0 FOLLOWER]: Starting forced leader election (received explicit request) I0507 06:28:51.800894 3708 raft_consensus.cc:2886] T dd440bce55914e85a2058ea4a1592ef0 P 5855917eaa4d4a4c96f9c0e36124ea7c [term 0 FOLLOWER]: Advancing to term 1 W0507 06:28:51.907629 3621 tablet.cc:1786] T dd440bce55914e85a2058ea4a1592ef0 P c8068fde2be746779966e9a1037d779c: Can't schedule compaction. Clean time has not been advanced past its initial value. W0507 06:28:51.929150 3752 tablet.cc:1786] T dd440bce55914e85a2058ea4a1592ef0 P 5855917eaa4d4a4c96f9c0e36124ea7c: Can't schedule compaction. Clean time has not been advanced past its initial value. W0507 06:28:51.974984 3874 tablet.cc:1786] T dd440bce55914e85a2058ea4a1592ef0 P eeb9cd29752c4282b6955954cf29243a: Can't schedule compaction. Clean time has not been advanced past its initial value. I0507 06:28:52.345533 3497 heartbeater.cc:467] Master 127.0.57.254:39451 was elected leader, sending a full tablet report... W0507 06:28:52.595522 3708 consensus_meta.cc:220] T dd440bce55914e85a2058ea4a1592ef0 P 5855917eaa4d4a4c96f9c0e36124ea7c: Time spent flushing consensus metadata: real 0.794s user 0.002s sys 0.000s I0507 06:28:52.595688 3708 raft_consensus.cc:494] T dd440bce55914e85a2058ea4a1592ef0 P 5855917eaa4d4a4c96f9c0e36124ea7c [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeb9cd29752c4282b6955954cf29243a" member_type: VOTER last_known_addr { host: "127.0.57.196" port: 40737 } } peers { permanent_uuid: "c8068fde2be746779966e9a1037d779c" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 40627 } } peers { permanent_uuid: "5855917eaa4d4a4c96f9c0e36124ea7c" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 41035 } } I0507 06:28:52.597221 3708 leader_election.cc:296] T dd440bce55914e85a2058ea4a1592ef0 P 5855917eaa4d4a4c96f9c0e36124ea7c [CANDIDATE]: Term 1 election: Requested vote from peers eeb9cd29752c4282b6955954cf29243a (127.0.57.196:40737), c8068fde2be746779966e9a1037d779c (127.0.57.194:40627) I0507 06:28:52.606151 3830 tablet_service.cc:1286] Received RequestConsensusVote() RPC: tablet_id: "dd440bce55914e85a2058ea4a1592ef0" candidate_uuid: "5855917eaa4d4a4c96f9c0e36124ea7c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "eeb9cd29752c4282b6955954cf29243a" I0507 06:28:52.606469 3830 raft_consensus.cc:2886] T dd440bce55914e85a2058ea4a1592ef0 P eeb9cd29752c4282b6955954cf29243a [term 0 FOLLOWER]: Advancing to term 1 I0507 06:28:52.609066 3577 tablet_service.cc:1286] Received RequestConsensusVote() RPC: tablet_id: "dd440bce55914e85a2058ea4a1592ef0" candidate_uuid: "5855917eaa4d4a4c96f9c0e36124ea7c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "c8068fde2be746779966e9a1037d779c" I0507 06:28:52.609320 3577 raft_consensus.cc:2886] T dd440bce55914e85a2058ea4a1592ef0 P c8068fde2be746779966e9a1037d779c [term 0 FOLLOWER]: Advancing to term 1 W0507 06:28:52.780885 3642 leader_election.cc:341] T dd440bce55914e85a2058ea4a1592ef0 P 5855917eaa4d4a4c96f9c0e36124ea7c [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer eeb9cd29752c4282b6955954cf29243a (127.0.57.196:40737): Timed out: RequestConsensusVote RPC to 127.0.57.196:40737 timed out after 0.183s (SENT) W0507 06:28:52.781132 3639 leader_election.cc:341] T dd440bce55914e85a2058ea4a1592ef0 P 5855917eaa4d4a4c96f9c0e36124ea7c [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer c8068fde2be746779966e9a1037d779c (127.0.57.194:40627): Timed out: RequestConsensusVote RPC to 127.0.57.194:40627 timed out after 0.183s (SENT) I0507 06:28:52.781303 3639 leader_election.cc:310] T dd440bce55914e85a2058ea4a1592ef0 P 5855917eaa4d4a4c96f9c0e36124ea7c [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5855917eaa4d4a4c96f9c0e36124ea7c; no voters: c8068fde2be746779966e9a1037d779c, eeb9cd29752c4282b6955954cf29243a I0507 06:28:52.782310 3908 raft_consensus.cc:2588] T dd440bce55914e85a2058ea4a1592ef0 P 5855917eaa4d4a4c96f9c0e36124ea7c [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority W0507 06:28:53.334300 3577 consensus_meta.cc:220] T dd440bce55914e85a2058ea4a1592ef0 P c8068fde2be746779966e9a1037d779c: Time spent flushing consensus metadata: real 0.725s user 0.002s sys 0.000s W0507 06:28:53.334378 3830 consensus_meta.cc:220] T dd440bce55914e85a2058ea4a1592ef0 P eeb9cd29752c4282b6955954cf29243a: Time spent flushing consensus metadata: real 0.728s user 0.003s sys 0.000s I0507 06:28:53.334429 3577 raft_consensus.cc:2346] T dd440bce55914e85a2058ea4a1592ef0 P c8068fde2be746779966e9a1037d779c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5855917eaa4d4a4c96f9c0e36124ea7c in term 1. I0507 06:28:53.334493 3830 raft_consensus.cc:2346] T dd440bce55914e85a2058ea4a1592ef0 P eeb9cd29752c4282b6955954cf29243a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5855917eaa4d4a4c96f9c0e36124ea7c in term 1. W0507 06:28:53.334692 3577 rpcz_store.cc:253] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.57.195:58359 (request call id 1) took 726 ms (726 ms). Client timeout 183 ms (183 ms) W0507 06:28:53.334786 3830 rpcz_store.cc:253] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.57.195:57567 (request call id 1) took 729 ms (729 ms). Client timeout 183 ms (183 ms) W0507 06:28:53.335321 3577 rpcz_store.cc:259] Trace: 0507 06:28:52.608604 (+ 0us) service_pool.cc:162] Inserting onto call queue 0507 06:28:52.608678 (+ 74us) service_pool.cc:221] Handling call 0507 06:28:53.334631 (+725953us) inbound_call.cc:162] Queueing success response Metrics: {} W0507 06:28:53.335347 3830 rpcz_store.cc:259] Trace: 0507 06:28:52.605603 (+ 0us) service_pool.cc:162] Inserting onto call queue 0507 06:28:52.605700 (+ 97us) service_pool.cc:221] Handling call 0507 06:28:53.334709 (+729009us) inbound_call.cc:162] Queueing success response Metrics: {} I0507 06:30:50.957087 3332 maintenance_manager.cc:306] P 7c82905e75974072bf33c7f4986bbc1a: Scheduling FlushMRSOp(00000000000000000000000000000000): perf score=0.033377 I0507 06:30:50.982837 3331 maintenance_manager.cc:501] P 7c82905e75974072bf33c7f4986bbc1a: FlushMRSOp(00000000000000000000000000000000) complete. Timing: real 0.025s user 0.018s sys 0.004s Metrics: {"bytes_written":6725,"cfile_init":1,"data dirs.queue_time_us":460,"data dirs.run_cpu_time_us":214,"data dirs.run_wall_time_us":664,"drs_written":1,"lbm_read_time_us":72,"lbm_reads_lt_1ms":4,"lbm_write_time_us":446,"lbm_writes_lt_1ms":27,"rows_written":4,"thread_start_us":434,"threads_started":2,"wal-append.queue_time_us":303} I0507 06:30:50.984519 3332 maintenance_manager.cc:306] P 7c82905e75974072bf33c7f4986bbc1a: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 475 bytes on disk I0507 06:30:50.985410 3331 maintenance_manager.cc:501] P 7c82905e75974072bf33c7f4986bbc1a: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":82,"lbm_reads_lt_1ms":4} /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/raft_consensus_nonvoter-itest.cc:859: Failure Failed Bad status: Timed out: Replica TabletServer: 5855917eaa4d4a4c96f9c0e36124ea7c, Rpc address: host: "127.0.57.195" port: 41035 is not leader after waiting for 120.000s: Not found: Error connecting to replica: Timed out: GetConsensusState RPC to 127.0.57.195:41035 timed out after -0.121s (ON_OUTBOUND_QUEUE) I0507 06:30:52.723107 231 external_mini_cluster.cc:1172] Attempting to check leaks for /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver pid 3377 I0507 06:30:52.731451 3494 generic_service.cc:171] Checking for leaks (request via RPC) I0507 06:30:52.788933 231 external_mini_cluster.cc:1131] Killing /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver with pid 3377 I0507 06:30:52.801383 231 external_mini_cluster.cc:1172] Attempting to check leaks for /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver pid 3501 I0507 06:30:52.809049 3617 generic_service.cc:171] Checking for leaks (request via RPC) I0507 06:30:52.866447 231 external_mini_cluster.cc:1131] Killing /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver with pid 3501 I0507 06:30:52.878007 231 external_mini_cluster.cc:1172] Attempting to check leaks for /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver pid 3623 I0507 06:30:52.883416 3748 generic_service.cc:171] Checking for leaks (request via RPC) I0507 06:30:52.946529 231 external_mini_cluster.cc:1131] Killing /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver with pid 3623 I0507 06:30:52.960642 231 external_mini_cluster.cc:1172] Attempting to check leaks for /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver pid 3754 I0507 06:30:52.966073 3870 generic_service.cc:171] Checking for leaks (request via RPC) I0507 06:30:53.025936 231 external_mini_cluster.cc:1131] Killing /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver with pid 3754 W0507 06:30:53.039599 3327 connection.cc:647] server connection from 127.0.57.196:38815 recv error: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107) I0507 06:30:53.039849 231 external_mini_cluster.cc:1172] Attempting to check leaks for /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-master pid 3317 I0507 06:30:53.045279 3373 generic_service.cc:171] Checking for leaks (request via RPC) I0507 06:30:53.146744 231 external_mini_cluster.cc:1131] Killing /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-master with pid 3317 I0507 06:30:53.170845 231 test_util.cc:135] ----------------------------------------------- I0507 06:30:53.170907 231 test_util.cc:136] Had fatal failures, leaving test files at /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromoteAndDemote.1557210500371863-231 [ FAILED ] RaftConsensusNonVoterITest.PromoteAndDemote (123429 ms) [ RUN ] RaftConsensusNonVoterITest.PromotedReplicaCanVote I0507 06:30:53.172696 231 test_util.cc:212] Using random seed: 1540531124 I0507 06:30:53.174216 231 ts_itest-base.cc:112] Starting cluster with: I0507 06:30:53.174268 231 ts_itest-base.cc:113] -------------- I0507 06:30:53.174295 231 ts_itest-base.cc:114] 4 tablet servers I0507 06:30:53.174336 231 ts_itest-base.cc:115] 3 replicas per TS I0507 06:30:53.174365 231 ts_itest-base.cc:116] -------------- I0507 06:30:53.175671 231 external_mini_cluster.cc:933] Running /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-master /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-master --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromotedReplicaCanVote.1557210500371863-231/raft_consensus-itest-cluster/master-0/wal --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromotedReplicaCanVote.1557210500371863-231/raft_consensus-itest-cluster/master-0/data --block_manager=log --webserver_interface=localhost --ipki_ca_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.0.57.254:44959 --webserver_interface=127.0.57.254 --webserver_port=0 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromotedReplicaCanVote.1557210500371863-231/raft_consensus-itest-cluster/master-0/logs --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromotedReplicaCanVote.1557210500371863-231/raft_consensus-itest-cluster/master-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --catalog_manager_evict_excess_replicas=false --rpc_reuseport=true with env {} WARNING: Logging before InitGoogleLogging() is written to STDERR W0507 06:30:53.391750 3927 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0507 06:30:53.392026 3927 flags.cc:404] Enabled unsafe flag: --never_fsync=true W0507 06:30:53.394587 3927 flags.cc:404] Enabled experimental flag: --ipki_ca_key_size=1024 W0507 06:30:53.394641 3927 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W0507 06:30:53.394672 3927 flags.cc:404] Enabled experimental flag: --tsk_num_rsa_bits=512 W0507 06:30:53.394697 3927 flags.cc:404] Enabled experimental flag: --rpc_reuseport=true I0507 06:30:53.399550 3927 master_main.cc:73] Master server non-default flags: --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromotedReplicaCanVote.1557210500371863-231/raft_consensus-itest-cluster/master-0/data --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromotedReplicaCanVote.1557210500371863-231/raft_consensus-itest-cluster/master-0/wal --catalog_manager_evict_excess_replicas=false --ipki_ca_key_size=1024 --ipki_server_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.0.57.254:44959 --rpc_reuseport=true --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromotedReplicaCanVote.1557210500371863-231/raft_consensus-itest-cluster/master-0/data/info.pb --webserver_interface=127.0.57.254 --webserver_port=0 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromotedReplicaCanVote.1557210500371863-231/raft_consensus-itest-cluster/master-0/logs --logbuflevel=-1 --logtostderr=true Master server version: kudu 1.10.0-cdh6.x-SNAPSHOT revision 7d0ee26697926e87af9ef395008799577f6ab43f build type FASTDEBUG built by jenkins at 06 May 2019 22:44:21 PST on quasar-knmpfx-1.vpc.cloudera.com build id 322017 ASAN enabled I0507 06:30:53.407347 3927 system_ntp.cc:202] NTP initialized. Skew: 500ppm Current error: 101016us I0507 06:30:53.408901 3927 fs_manager.cc:263] Metadata directory not provided I0507 06:30:53.408995 3927 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0507 06:30:53.409332 3927 server_base.cc:433] Could not load existing FS layout: Not found: could not find a healthy instance file I0507 06:30:53.409418 3927 server_base.cc:434] Attempting to create new FS layout instead I0507 06:30:53.412770 3927 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromotedReplicaCanVote.1557210500371863-231/raft_consensus-itest-cluster/master-0/data/instance: uuid: "bc4d61e0b69f468f97daed3cf6089555" format_stamp: "Formatted at 2019-05-07 06:30:53 on dist-test-slave-dist-test-slave-rqp6" I0507 06:30:53.413362 3927 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromotedReplicaCanVote.1557210500371863-231/raft_consensus-itest-cluster/master-0/wal/instance: uuid: "bc4d61e0b69f468f97daed3cf6089555" format_stamp: "Formatted at 2019-05-07 06:30:53 on dist-test-slave-dist-test-slave-rqp6" I0507 06:30:53.421454 3927 fs_manager.cc:503] Time spent creating directory manager: real 0.008s user 0.006s sys 0.001s I0507 06:30:53.422782 3927 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0507 06:30:53.423454 3927 file_cache.cc:466] Constructed file cache lbm with capacity 419430 I0507 06:30:53.427255 3927 fs_manager.cc:419] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s I0507 06:30:53.427335 3927 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromotedReplicaCanVote.1557210500371863-231/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromotedReplicaCanVote.1557210500371863-231/raft_consensus-itest-cluster/master-0/wal uuid: "bc4d61e0b69f468f97daed3cf6089555" format_stamp: "Formatted at 2019-05-07 06:30:53 on dist-test-slave-dist-test-slave-rqp6" I0507 06:30:53.427505 3927 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromotedReplicaCanVote.1557210500371863-231/raft_consensus-itest-cluster/master-0/wal metadata directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromotedReplicaCanVote.1557210500371863-231/raft_consensus-itest-cluster/master-0/wal 1 data directories: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromotedReplicaCanVote.1557210500371863-231/raft_consensus-itest-cluster/master-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0507 06:30:53.467931 3927 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0507 06:30:53.483248 3927 rpc_server.cc:205] RPC server started. Bound to: 127.0.57.254:44959 I0507 06:30:53.484575 3927 webserver.cc:308] Webserver started at http://127.0.57.254:45249/ using document root and password file I0507 06:30:53.485450 3927 server_base.cc:608] Dumped server information to /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromotedReplicaCanVote.1557210500371863-231/raft_consensus-itest-cluster/master-0/data/info.pb I0507 06:30:53.489497 3986 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0507 06:30:53.492919 231 external_mini_cluster.cc:995] Started /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-master as pid 3927 I0507 06:30:53.493652 231 external_mini_cluster.cc:933] Running /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromotedReplicaCanVote.1557210500371863-231/raft_consensus-itest-cluster/ts-0/wal --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromotedReplicaCanVote.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data --block_manager=log --rpc_bind_addresses=127.0.57.193:0 --local_ip_for_outbound_sockets=127.0.57.193 --webserver_interface=127.0.57.193 --webserver_port=0 --tserver_master_addrs=127.0.57.254:44959 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromotedReplicaCanVote.1557210500371863-231/raft_consensus-itest-cluster/ts-0/logs --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromotedReplicaCanVote.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_heartbeat_interval_ms=50 --evict_failed_followers=false with env {} I0507 06:30:53.505100 3986 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P bc4d61e0b69f468f97daed3cf6089555: Bootstrap starting. I0507 06:30:53.508275 3986 tablet_bootstrap.cc:590] T 00000000000000000000000000000000 P bc4d61e0b69f468f97daed3cf6089555: No blocks or log segments found. Creating new log. I0507 06:30:53.513357 3986 log.cc:525] T 00000000000000000000000000000000 P bc4d61e0b69f468f97daed3cf6089555: Log is configured to *not* fsync() on all Append() calls I0507 06:30:53.518306 3986 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P bc4d61e0b69f468f97daed3cf6089555: No bootstrap required, opened a new log I0507 06:30:53.525226 3986 raft_consensus.cc:340] T 00000000000000000000000000000000 P bc4d61e0b69f468f97daed3cf6089555 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc4d61e0b69f468f97daed3cf6089555" member_type: VOTER } I0507 06:30:53.525485 3986 raft_consensus.cc:366] T 00000000000000000000000000000000 P bc4d61e0b69f468f97daed3cf6089555 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0507 06:30:53.525557 3986 raft_consensus.cc:705] T 00000000000000000000000000000000 P bc4d61e0b69f468f97daed3cf6089555 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc4d61e0b69f468f97daed3cf6089555, State: Initialized, Role: FOLLOWER I0507 06:30:53.526111 3986 consensus_queue.cc:229] T 00000000000000000000000000000000 P bc4d61e0b69f468f97daed3cf6089555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc4d61e0b69f468f97daed3cf6089555" member_type: VOTER } I0507 06:30:53.526290 3986 raft_consensus.cc:380] T 00000000000000000000000000000000 P bc4d61e0b69f468f97daed3cf6089555 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0507 06:30:53.526355 3986 raft_consensus.cc:472] T 00000000000000000000000000000000 P bc4d61e0b69f468f97daed3cf6089555 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0507 06:30:53.526440 3986 raft_consensus.cc:2886] T 00000000000000000000000000000000 P bc4d61e0b69f468f97daed3cf6089555 [term 0 FOLLOWER]: Advancing to term 1 I0507 06:30:53.528139 3986 raft_consensus.cc:494] T 00000000000000000000000000000000 P bc4d61e0b69f468f97daed3cf6089555 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc4d61e0b69f468f97daed3cf6089555" member_type: VOTER } I0507 06:30:53.528636 3986 leader_election.cc:310] T 00000000000000000000000000000000 P bc4d61e0b69f468f97daed3cf6089555 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bc4d61e0b69f468f97daed3cf6089555; no voters: I0507 06:30:53.532738 3986 leader_election.cc:296] T 00000000000000000000000000000000 P bc4d61e0b69f468f97daed3cf6089555 [CANDIDATE]: Term 1 election: Requested vote from peers I0507 06:30:53.533156 3989 raft_consensus.cc:2640] T 00000000000000000000000000000000 P bc4d61e0b69f468f97daed3cf6089555 [term 1 FOLLOWER]: Leader election won for term 1 I0507 06:30:53.534250 3989 raft_consensus.cc:667] T 00000000000000000000000000000000 P bc4d61e0b69f468f97daed3cf6089555 [term 1 LEADER]: Becoming Leader. State: Replica: bc4d61e0b69f468f97daed3cf6089555, State: Running, Role: LEADER I0507 06:30:53.534803 3989 consensus_queue.cc:206] T 00000000000000000000000000000000 P bc4d61e0b69f468f97daed3cf6089555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc4d61e0b69f468f97daed3cf6089555" member_type: VOTER } I0507 06:30:53.537207 3990 sys_catalog.cc:337] T 00000000000000000000000000000000 P bc4d61e0b69f468f97daed3cf6089555 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bc4d61e0b69f468f97daed3cf6089555" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc4d61e0b69f468f97daed3cf6089555" member_type: VOTER } } I0507 06:30:53.537431 3986 sys_catalog.cc:337] T 00000000000000000000000000000000 P bc4d61e0b69f468f97daed3cf6089555 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 1 leader_uuid: "bc4d61e0b69f468f97daed3cf6089555" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc4d61e0b69f468f97daed3cf6089555" member_type: VOTER } } I0507 06:30:53.537816 3990 sys_catalog.cc:340] T 00000000000000000000000000000000 P bc4d61e0b69f468f97daed3cf6089555 [sys.catalog]: This master's current role is: LEADER I0507 06:30:53.537907 3986 sys_catalog.cc:340] T 00000000000000000000000000000000 P bc4d61e0b69f468f97daed3cf6089555 [sys.catalog]: This master's current role is: LEADER I0507 06:30:53.537223 3991 sys_catalog.cc:337] T 00000000000000000000000000000000 P bc4d61e0b69f468f97daed3cf6089555 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bc4d61e0b69f468f97daed3cf6089555. Latest consensus state: current_term: 1 leader_uuid: "bc4d61e0b69f468f97daed3cf6089555" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc4d61e0b69f468f97daed3cf6089555" member_type: VOTER } } I0507 06:30:53.538273 3991 sys_catalog.cc:340] T 00000000000000000000000000000000 P bc4d61e0b69f468f97daed3cf6089555 [sys.catalog]: This master's current role is: LEADER I0507 06:30:53.539481 3986 sys_catalog.cc:424] T 00000000000000000000000000000000 P bc4d61e0b69f468f97daed3cf6089555 [sys.catalog]: configured and running, proceeding with master startup. I0507 06:30:53.539683 3993 catalog_manager.cc:1055] Loading table and tablet metadata into memory... I0507 06:30:53.550868 3993 catalog_manager.cc:1066] Initializing Kudu internal certificate authority... I0507 06:30:53.578925 3993 catalog_manager.cc:950] Generated new certificate authority record I0507 06:30:53.580507 3993 catalog_manager.cc:1075] Loading token signing keys... I0507 06:30:53.595144 3993 catalog_manager.cc:4292] T 00000000000000000000000000000000 P bc4d61e0b69f468f97daed3cf6089555: Generated new TSK 0 WARNING: Logging before InitGoogleLogging() is written to STDERR W0507 06:30:53.665781 3987 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0507 06:30:53.666004 3987 flags.cc:404] Enabled unsafe flag: --never_fsync=true W0507 06:30:53.668056 3987 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W0507 06:30:53.668155 3987 flags.cc:404] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.57.193 I0507 06:30:53.684762 3987 tablet_server_main.cc:78] Tablet server non-default flags: --evict_failed_followers=false --raft_heartbeat_interval_ms=50 --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromotedReplicaCanVote.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromotedReplicaCanVote.1557210500371863-231/raft_consensus-itest-cluster/ts-0/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.0.57.193:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromotedReplicaCanVote.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data/info.pb --webserver_interface=127.0.57.193 --webserver_port=0 --tserver_master_addrs=127.0.57.254:44959 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.0.57.193 --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromotedReplicaCanVote.1557210500371863-231/raft_consensus-itest-cluster/ts-0/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.10.0-cdh6.x-SNAPSHOT revision 7d0ee26697926e87af9ef395008799577f6ab43f build type FASTDEBUG built by jenkins at 06 May 2019 22:44:21 PST on quasar-knmpfx-1.vpc.cloudera.com build id 322017 ASAN enabled I0507 06:30:53.692750 3987 system_ntp.cc:202] NTP initialized. Skew: 500ppm Current error: 101016us I0507 06:30:53.693419 3987 fs_manager.cc:263] Metadata directory not provided I0507 06:30:53.693495 3987 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0507 06:30:53.693749 3987 server_base.cc:433] Could not load existing FS layout: Not found: could not find a healthy instance file I0507 06:30:53.693825 3987 server_base.cc:434] Attempting to create new FS layout instead I0507 06:30:53.696277 3987 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromotedReplicaCanVote.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data/instance: uuid: "6f2066e97dec4d75a86e72155048de14" format_stamp: "Formatted at 2019-05-07 06:30:53 on dist-test-slave-dist-test-slave-rqp6" I0507 06:30:53.696837 3987 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromotedReplicaCanVote.1557210500371863-231/raft_consensus-itest-cluster/ts-0/wal/instance: uuid: "6f2066e97dec4d75a86e72155048de14" format_stamp: "Formatted at 2019-05-07 06:30:53 on dist-test-slave-dist-test-slave-rqp6" I0507 06:30:53.714877 3987 fs_manager.cc:503] Time spent creating directory manager: real 0.018s user 0.011s sys 0.005s I0507 06:30:53.716187 3987 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0507 06:30:53.716840 3987 file_cache.cc:466] Constructed file cache lbm with capacity 419430 I0507 06:30:53.720422 3987 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.000s sys 0.000s I0507 06:30:53.720507 3987 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromotedReplicaCanVote.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromotedReplicaCanVote.1557210500371863-231/raft_consensus-itest-cluster/ts-0/wal uuid: "6f2066e97dec4d75a86e72155048de14" format_stamp: "Formatted at 2019-05-07 06:30:53 on dist-test-slave-dist-test-slave-rqp6" I0507 06:30:53.720727 3987 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromotedReplicaCanVote.1557210500371863-231/raft_consensus-itest-cluster/ts-0/wal metadata directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromotedReplicaCanVote.1557210500371863-231/raft_consensus-itest-cluster/ts-0/wal 1 data directories: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromotedReplicaCanVote.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0507 06:30:53.765045 3987 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0507 06:30:53.769614 3987 ts_tablet_manager.cc:356] Loaded tablet metadata (0 total tablets, 0 live tablets) I0507 06:30:53.769691 3987 ts_tablet_manager.cc:376] Registered 0 tablets I0507 06:30:53.864832 3987 rpc_server.cc:205] RPC server started. Bound to: 127.0.57.193:43041 I0507 06:30:53.866050 3987 webserver.cc:308] Webserver started at http://127.0.57.193:45245/ using document root and password file I0507 06:30:53.866827 3987 server_base.cc:608] Dumped server information to /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromotedReplicaCanVote.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data/info.pb I0507 06:30:53.869582 231 external_mini_cluster.cc:995] Started /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver as pid 3987 I0507 06:30:53.870201 231 external_mini_cluster.cc:933] Running /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromotedReplicaCanVote.1557210500371863-231/raft_consensus-itest-cluster/ts-1/wal --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromotedReplicaCanVote.1557210500371863-231/raft_consensus-itest-cluster/ts-1/data --block_manager=log --rpc_bind_addresses=127.0.57.194:0 --local_ip_for_outbound_sockets=127.0.57.194 --webserver_interface=127.0.57.194 --webserver_port=0 --tserver_master_addrs=127.0.57.254:44959 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromotedReplicaCanVote.1557210500371863-231/raft_consensus-itest-cluster/ts-1/logs --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromotedReplicaCanVote.1557210500371863-231/raft_consensus-itest-cluster/ts-1/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_heartbeat_interval_ms=50 --evict_failed_followers=false with env {} I0507 06:30:53.892375 4116 heartbeater.cc:345] Connected to a master server at 127.0.57.254:44959 I0507 06:30:53.892619 4116 heartbeater.cc:426] Registering TS with master... I0507 06:30:53.893182 4116 heartbeater.cc:475] Master 127.0.57.254:44959 requested a full tablet report, sending... I0507 06:30:53.895071 3953 ts_manager.cc:159] Registered new tserver with Master: 6f2066e97dec4d75a86e72155048de14 (127.0.57.193:43041) I0507 06:30:53.896801 3953 master_service.cc:273] Signed X509 certificate for tserver {username='slave'} at 127.0.57.193:44209 WARNING: Logging before InitGoogleLogging() is written to STDERR W0507 06:30:54.036036 4120 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0507 06:30:54.036257 4120 flags.cc:404] Enabled unsafe flag: --never_fsync=true W0507 06:30:54.039172 4120 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W0507 06:30:54.039278 4120 flags.cc:404] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.57.194 I0507 06:30:54.043817 4120 tablet_server_main.cc:78] Tablet server non-default flags: --evict_failed_followers=false --raft_heartbeat_interval_ms=50 --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromotedReplicaCanVote.1557210500371863-231/raft_consensus-itest-cluster/ts-1/data --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromotedReplicaCanVote.1557210500371863-231/raft_consensus-itest-cluster/ts-1/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.0.57.194:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromotedReplicaCanVote.1557210500371863-231/raft_consensus-itest-cluster/ts-1/data/info.pb --webserver_interface=127.0.57.194 --webserver_port=0 --tserver_master_addrs=127.0.57.254:44959 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.0.57.194 --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromotedReplicaCanVote.1557210500371863-231/raft_consensus-itest-cluster/ts-1/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.10.0-cdh6.x-SNAPSHOT revision 7d0ee26697926e87af9ef395008799577f6ab43f build type FASTDEBUG built by jenkins at 06 May 2019 22:44:21 PST on quasar-knmpfx-1.vpc.cloudera.com build id 322017 ASAN enabled I0507 06:30:54.051903 4120 system_ntp.cc:202] NTP initialized. Skew: 500ppm Current error: 101516us I0507 06:30:54.052578 4120 fs_manager.cc:263] Metadata directory not provided I0507 06:30:54.052654 4120 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0507 06:30:54.052913 4120 server_base.cc:433] Could not load existing FS layout: Not found: could not find a healthy instance file I0507 06:30:54.053086 4120 server_base.cc:434] Attempting to create new FS layout instead I0507 06:30:54.055635 4120 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromotedReplicaCanVote.1557210500371863-231/raft_consensus-itest-cluster/ts-1/data/instance: uuid: "ae72da07d97b4d9b9e0d246ff83901ed" format_stamp: "Formatted at 2019-05-07 06:30:54 on dist-test-slave-dist-test-slave-rqp6" I0507 06:30:54.056304 4120 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromotedReplicaCanVote.1557210500371863-231/raft_consensus-itest-cluster/ts-1/wal/instance: uuid: "ae72da07d97b4d9b9e0d246ff83901ed" format_stamp: "Formatted at 2019-05-07 06:30:54 on dist-test-slave-dist-test-slave-rqp6" I0507 06:30:54.063083 4120 fs_manager.cc:503] Time spent creating directory manager: real 0.007s user 0.009s sys 0.000s I0507 06:30:54.064185 4120 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0507 06:30:54.064766 4120 file_cache.cc:466] Constructed file cache lbm with capacity 419430 I0507 06:30:54.068356 4120 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s I0507 06:30:54.068431 4120 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromotedReplicaCanVote.1557210500371863-231/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromotedReplicaCanVote.1557210500371863-231/raft_consensus-itest-cluster/ts-1/wal uuid: "ae72da07d97b4d9b9e0d246ff83901ed" format_stamp: "Formatted at 2019-05-07 06:30:54 on dist-test-slave-dist-test-slave-rqp6" I0507 06:30:54.068629 4120 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromotedReplicaCanVote.1557210500371863-231/raft_consensus-itest-cluster/ts-1/wal metadata directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromotedReplicaCanVote.1557210500371863-231/raft_consensus-itest-cluster/ts-1/wal 1 data directories: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromotedReplicaCanVote.1557210500371863-231/raft_consensus-itest-cluster/ts-1/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0507 06:30:54.097437 4120 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0507 06:30:54.099349 4120 ts_tablet_manager.cc:356] Loaded tablet metadata (0 total tablets, 0 live tablets) I0507 06:30:54.099414 4120 ts_tablet_manager.cc:376] Registered 0 tablets I0507 06:30:54.127097 4120 rpc_server.cc:205] RPC server started. Bound to: 127.0.57.194:34185 I0507 06:30:54.128190 4120 webserver.cc:308] Webserver started at http://127.0.57.194:33973/ using document root and password file I0507 06:30:54.128893 4120 server_base.cc:608] Dumped server information to /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromotedReplicaCanVote.1557210500371863-231/raft_consensus-itest-cluster/ts-1/data/info.pb I0507 06:30:54.132181 231 external_mini_cluster.cc:995] Started /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver as pid 4120 I0507 06:30:54.132930 231 external_mini_cluster.cc:933] Running /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromotedReplicaCanVote.1557210500371863-231/raft_consensus-itest-cluster/ts-2/wal --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromotedReplicaCanVote.1557210500371863-231/raft_consensus-itest-cluster/ts-2/data --block_manager=log --rpc_bind_addresses=127.0.57.195:0 --local_ip_for_outbound_sockets=127.0.57.195 --webserver_interface=127.0.57.195 --webserver_port=0 --tserver_master_addrs=127.0.57.254:44959 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromotedReplicaCanVote.1557210500371863-231/raft_consensus-itest-cluster/ts-2/logs --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromotedReplicaCanVote.1557210500371863-231/raft_consensus-itest-cluster/ts-2/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_heartbeat_interval_ms=50 --evict_failed_followers=false with env {} I0507 06:30:54.142534 4239 heartbeater.cc:345] Connected to a master server at 127.0.57.254:44959 I0507 06:30:54.142776 4239 heartbeater.cc:426] Registering TS with master... I0507 06:30:54.143407 4239 heartbeater.cc:475] Master 127.0.57.254:44959 requested a full tablet report, sending... I0507 06:30:54.144573 3953 ts_manager.cc:159] Registered new tserver with Master: ae72da07d97b4d9b9e0d246ff83901ed (127.0.57.194:34185) I0507 06:30:54.145498 3953 master_service.cc:273] Signed X509 certificate for tserver {username='slave'} at 127.0.57.194:41379 WARNING: Logging before InitGoogleLogging() is written to STDERR W0507 06:30:54.294909 4242 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0507 06:30:54.295150 4242 flags.cc:404] Enabled unsafe flag: --never_fsync=true W0507 06:30:54.299639 4242 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W0507 06:30:54.299754 4242 flags.cc:404] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.57.195 I0507 06:30:54.304381 4242 tablet_server_main.cc:78] Tablet server non-default flags: --evict_failed_followers=false --raft_heartbeat_interval_ms=50 --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromotedReplicaCanVote.1557210500371863-231/raft_consensus-itest-cluster/ts-2/data --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromotedReplicaCanVote.1557210500371863-231/raft_consensus-itest-cluster/ts-2/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.0.57.195:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromotedReplicaCanVote.1557210500371863-231/raft_consensus-itest-cluster/ts-2/data/info.pb --webserver_interface=127.0.57.195 --webserver_port=0 --tserver_master_addrs=127.0.57.254:44959 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.0.57.195 --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromotedReplicaCanVote.1557210500371863-231/raft_consensus-itest-cluster/ts-2/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.10.0-cdh6.x-SNAPSHOT revision 7d0ee26697926e87af9ef395008799577f6ab43f build type FASTDEBUG built by jenkins at 06 May 2019 22:44:21 PST on quasar-knmpfx-1.vpc.cloudera.com build id 322017 ASAN enabled I0507 06:30:54.311899 4242 system_ntp.cc:202] NTP initialized. Skew: 500ppm Current error: 101516us I0507 06:30:54.312561 4242 fs_manager.cc:263] Metadata directory not provided I0507 06:30:54.312638 4242 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0507 06:30:54.312882 4242 server_base.cc:433] Could not load existing FS layout: Not found: could not find a healthy instance file I0507 06:30:54.312955 4242 server_base.cc:434] Attempting to create new FS layout instead I0507 06:30:54.315269 4242 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromotedReplicaCanVote.1557210500371863-231/raft_consensus-itest-cluster/ts-2/data/instance: uuid: "7eecdaa8d1dc4a9988492494fbb17f1d" format_stamp: "Formatted at 2019-05-07 06:30:54 on dist-test-slave-dist-test-slave-rqp6" I0507 06:30:54.315814 4242 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromotedReplicaCanVote.1557210500371863-231/raft_consensus-itest-cluster/ts-2/wal/instance: uuid: "7eecdaa8d1dc4a9988492494fbb17f1d" format_stamp: "Formatted at 2019-05-07 06:30:54 on dist-test-slave-dist-test-slave-rqp6" I0507 06:30:54.334986 4242 fs_manager.cc:503] Time spent creating directory manager: real 0.019s user 0.007s sys 0.013s I0507 06:30:54.336285 4242 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0507 06:30:54.336977 4242 file_cache.cc:466] Constructed file cache lbm with capacity 419430 I0507 06:30:54.340452 4242 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.000s sys 0.000s I0507 06:30:54.340548 4242 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromotedReplicaCanVote.1557210500371863-231/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromotedReplicaCanVote.1557210500371863-231/raft_consensus-itest-cluster/ts-2/wal uuid: "7eecdaa8d1dc4a9988492494fbb17f1d" format_stamp: "Formatted at 2019-05-07 06:30:54 on dist-test-slave-dist-test-slave-rqp6" I0507 06:30:54.340728 4242 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromotedReplicaCanVote.1557210500371863-231/raft_consensus-itest-cluster/ts-2/wal metadata directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromotedReplicaCanVote.1557210500371863-231/raft_consensus-itest-cluster/ts-2/wal 1 data directories: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromotedReplicaCanVote.1557210500371863-231/raft_consensus-itest-cluster/ts-2/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0507 06:30:54.417286 4242 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0507 06:30:54.419000 4242 ts_tablet_manager.cc:356] Loaded tablet metadata (0 total tablets, 0 live tablets) I0507 06:30:54.419068 4242 ts_tablet_manager.cc:376] Registered 0 tablets I0507 06:30:54.446087 4242 rpc_server.cc:205] RPC server started. Bound to: 127.0.57.195:33833 I0507 06:30:54.447340 4242 webserver.cc:308] Webserver started at http://127.0.57.195:43957/ using document root and password file I0507 06:30:54.448112 4242 server_base.cc:608] Dumped server information to /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromotedReplicaCanVote.1557210500371863-231/raft_consensus-itest-cluster/ts-2/data/info.pb I0507 06:30:54.453881 231 external_mini_cluster.cc:995] Started /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver as pid 4242 I0507 06:30:54.454555 231 external_mini_cluster.cc:933] Running /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromotedReplicaCanVote.1557210500371863-231/raft_consensus-itest-cluster/ts-3/wal --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromotedReplicaCanVote.1557210500371863-231/raft_consensus-itest-cluster/ts-3/data --block_manager=log --rpc_bind_addresses=127.0.57.196:0 --local_ip_for_outbound_sockets=127.0.57.196 --webserver_interface=127.0.57.196 --webserver_port=0 --tserver_master_addrs=127.0.57.254:44959 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromotedReplicaCanVote.1557210500371863-231/raft_consensus-itest-cluster/ts-3/logs --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromotedReplicaCanVote.1557210500371863-231/raft_consensus-itest-cluster/ts-3/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_heartbeat_interval_ms=50 --evict_failed_followers=false with env {} I0507 06:30:54.461773 4361 heartbeater.cc:345] Connected to a master server at 127.0.57.254:44959 I0507 06:30:54.462019 4361 heartbeater.cc:426] Registering TS with master... I0507 06:30:54.462620 4361 heartbeater.cc:475] Master 127.0.57.254:44959 requested a full tablet report, sending... I0507 06:30:54.463896 3953 ts_manager.cc:159] Registered new tserver with Master: 7eecdaa8d1dc4a9988492494fbb17f1d (127.0.57.195:33833) I0507 06:30:54.464910 3953 master_service.cc:273] Signed X509 certificate for tserver {username='slave'} at 127.0.57.195:44189 WARNING: Logging before InitGoogleLogging() is written to STDERR W0507 06:30:54.626688 4364 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0507 06:30:54.626937 4364 flags.cc:404] Enabled unsafe flag: --never_fsync=true W0507 06:30:54.629108 4364 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W0507 06:30:54.629221 4364 flags.cc:404] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.57.196 I0507 06:30:54.633322 4364 tablet_server_main.cc:78] Tablet server non-default flags: --evict_failed_followers=false --raft_heartbeat_interval_ms=50 --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromotedReplicaCanVote.1557210500371863-231/raft_consensus-itest-cluster/ts-3/data --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromotedReplicaCanVote.1557210500371863-231/raft_consensus-itest-cluster/ts-3/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.0.57.196:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromotedReplicaCanVote.1557210500371863-231/raft_consensus-itest-cluster/ts-3/data/info.pb --webserver_interface=127.0.57.196 --webserver_port=0 --tserver_master_addrs=127.0.57.254:44959 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.0.57.196 --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromotedReplicaCanVote.1557210500371863-231/raft_consensus-itest-cluster/ts-3/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.10.0-cdh6.x-SNAPSHOT revision 7d0ee26697926e87af9ef395008799577f6ab43f build type FASTDEBUG built by jenkins at 06 May 2019 22:44:21 PST on quasar-knmpfx-1.vpc.cloudera.com build id 322017 ASAN enabled I0507 06:30:54.641541 4364 system_ntp.cc:202] NTP initialized. Skew: 500ppm Current error: 101516us I0507 06:30:54.642307 4364 fs_manager.cc:263] Metadata directory not provided I0507 06:30:54.642388 4364 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0507 06:30:54.642674 4364 server_base.cc:433] Could not load existing FS layout: Not found: could not find a healthy instance file I0507 06:30:54.642757 4364 server_base.cc:434] Attempting to create new FS layout instead I0507 06:30:54.645565 4364 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromotedReplicaCanVote.1557210500371863-231/raft_consensus-itest-cluster/ts-3/data/instance: uuid: "a49a0f0a6258477d8126ad3c88bf0d44" format_stamp: "Formatted at 2019-05-07 06:30:54 on dist-test-slave-dist-test-slave-rqp6" I0507 06:30:54.646209 4364 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromotedReplicaCanVote.1557210500371863-231/raft_consensus-itest-cluster/ts-3/wal/instance: uuid: "a49a0f0a6258477d8126ad3c88bf0d44" format_stamp: "Formatted at 2019-05-07 06:30:54 on dist-test-slave-dist-test-slave-rqp6" I0507 06:30:54.653467 4364 fs_manager.cc:503] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s I0507 06:30:54.654613 4364 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0507 06:30:54.655254 4364 file_cache.cc:466] Constructed file cache lbm with capacity 419430 I0507 06:30:54.658439 4364 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s I0507 06:30:54.658509 4364 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromotedReplicaCanVote.1557210500371863-231/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromotedReplicaCanVote.1557210500371863-231/raft_consensus-itest-cluster/ts-3/wal uuid: "a49a0f0a6258477d8126ad3c88bf0d44" format_stamp: "Formatted at 2019-05-07 06:30:54 on dist-test-slave-dist-test-slave-rqp6" I0507 06:30:54.658694 4364 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromotedReplicaCanVote.1557210500371863-231/raft_consensus-itest-cluster/ts-3/wal metadata directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromotedReplicaCanVote.1557210500371863-231/raft_consensus-itest-cluster/ts-3/wal 1 data directories: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromotedReplicaCanVote.1557210500371863-231/raft_consensus-itest-cluster/ts-3/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0507 06:30:54.718489 4364 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0507 06:30:54.720360 4364 ts_tablet_manager.cc:356] Loaded tablet metadata (0 total tablets, 0 live tablets) I0507 06:30:54.720424 4364 ts_tablet_manager.cc:376] Registered 0 tablets I0507 06:30:54.746524 4364 rpc_server.cc:205] RPC server started. Bound to: 127.0.57.196:43761 I0507 06:30:54.747663 4364 webserver.cc:308] Webserver started at http://127.0.57.196:42933/ using document root and password file I0507 06:30:54.748383 4364 server_base.cc:608] Dumped server information to /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromotedReplicaCanVote.1557210500371863-231/raft_consensus-itest-cluster/ts-3/data/info.pb I0507 06:30:54.752312 231 external_mini_cluster.cc:995] Started /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver as pid 4364 I0507 06:30:54.762028 4483 heartbeater.cc:345] Connected to a master server at 127.0.57.254:44959 I0507 06:30:54.762261 4483 heartbeater.cc:426] Registering TS with master... I0507 06:30:54.762897 4483 heartbeater.cc:475] Master 127.0.57.254:44959 requested a full tablet report, sending... I0507 06:30:54.764122 3953 ts_manager.cc:159] Registered new tserver with Master: a49a0f0a6258477d8126ad3c88bf0d44 (127.0.57.196:43761) I0507 06:30:54.765054 231 external_mini_cluster.cc:623] 4 TS(s) registered with all masters I0507 06:30:54.765259 3953 master_service.cc:273] Signed X509 certificate for tserver {username='slave'} at 127.0.57.196:41163 I0507 06:30:54.777310 3953 catalog_manager.cc:1415] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56678: name: "TestTable" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 3 split_rows_range_bounds { } partition_schema { range_schema { columns { name: "key" } } } I0507 06:30:54.807833 4176 tablet_service.cc:1016] Processing CreateTablet for tablet 143439bb7e50454aab47fae4ed3ad4ae (table=TestTable [id=4caeb95c912c426cba70c644bf9e33bf]), partition=RANGE (key) PARTITION UNBOUNDED I0507 06:30:54.808064 4420 tablet_service.cc:1016] Processing CreateTablet for tablet 143439bb7e50454aab47fae4ed3ad4ae (table=TestTable [id=4caeb95c912c426cba70c644bf9e33bf]), partition=RANGE (key) PARTITION UNBOUNDED I0507 06:30:54.808133 4298 tablet_service.cc:1016] Processing CreateTablet for tablet 143439bb7e50454aab47fae4ed3ad4ae (table=TestTable [id=4caeb95c912c426cba70c644bf9e33bf]), partition=RANGE (key) PARTITION UNBOUNDED I0507 06:30:54.808969 4176 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet 143439bb7e50454aab47fae4ed3ad4ae. 1 dirs total, 0 dirs full, 0 dirs failed I0507 06:30:54.809001 4298 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet 143439bb7e50454aab47fae4ed3ad4ae. 1 dirs total, 0 dirs full, 0 dirs failed I0507 06:30:54.809010 4420 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet 143439bb7e50454aab47fae4ed3ad4ae. 1 dirs total, 0 dirs full, 0 dirs failed I0507 06:30:54.818174 4503 tablet_bootstrap.cc:439] T 143439bb7e50454aab47fae4ed3ad4ae P a49a0f0a6258477d8126ad3c88bf0d44: Bootstrap starting. I0507 06:30:54.818639 4504 tablet_bootstrap.cc:439] T 143439bb7e50454aab47fae4ed3ad4ae P 7eecdaa8d1dc4a9988492494fbb17f1d: Bootstrap starting. I0507 06:30:54.818874 4505 tablet_bootstrap.cc:439] T 143439bb7e50454aab47fae4ed3ad4ae P ae72da07d97b4d9b9e0d246ff83901ed: Bootstrap starting. I0507 06:30:54.821310 4504 tablet_bootstrap.cc:590] T 143439bb7e50454aab47fae4ed3ad4ae P 7eecdaa8d1dc4a9988492494fbb17f1d: No blocks or log segments found. Creating new log. I0507 06:30:54.822324 4504 log.cc:525] T 143439bb7e50454aab47fae4ed3ad4ae P 7eecdaa8d1dc4a9988492494fbb17f1d: Log is configured to *not* fsync() on all Append() calls I0507 06:30:54.822682 4503 tablet_bootstrap.cc:590] T 143439bb7e50454aab47fae4ed3ad4ae P a49a0f0a6258477d8126ad3c88bf0d44: No blocks or log segments found. Creating new log. I0507 06:30:54.823602 4503 log.cc:525] T 143439bb7e50454aab47fae4ed3ad4ae P a49a0f0a6258477d8126ad3c88bf0d44: Log is configured to *not* fsync() on all Append() calls I0507 06:30:54.825281 4505 tablet_bootstrap.cc:590] T 143439bb7e50454aab47fae4ed3ad4ae P ae72da07d97b4d9b9e0d246ff83901ed: No blocks or log segments found. Creating new log. I0507 06:30:54.826110 4505 log.cc:525] T 143439bb7e50454aab47fae4ed3ad4ae P ae72da07d97b4d9b9e0d246ff83901ed: Log is configured to *not* fsync() on all Append() calls I0507 06:30:54.827591 4503 tablet_bootstrap.cc:439] T 143439bb7e50454aab47fae4ed3ad4ae P a49a0f0a6258477d8126ad3c88bf0d44: No bootstrap required, opened a new log I0507 06:30:54.827623 4504 tablet_bootstrap.cc:439] T 143439bb7e50454aab47fae4ed3ad4ae P 7eecdaa8d1dc4a9988492494fbb17f1d: No bootstrap required, opened a new log I0507 06:30:54.827806 4505 tablet_bootstrap.cc:439] T 143439bb7e50454aab47fae4ed3ad4ae P ae72da07d97b4d9b9e0d246ff83901ed: No bootstrap required, opened a new log I0507 06:30:54.827950 4503 ts_tablet_manager.cc:1053] T 143439bb7e50454aab47fae4ed3ad4ae P a49a0f0a6258477d8126ad3c88bf0d44: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.000s I0507 06:30:54.828050 4504 ts_tablet_manager.cc:1053] T 143439bb7e50454aab47fae4ed3ad4ae P 7eecdaa8d1dc4a9988492494fbb17f1d: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s I0507 06:30:54.828064 4505 ts_tablet_manager.cc:1053] T 143439bb7e50454aab47fae4ed3ad4ae P ae72da07d97b4d9b9e0d246ff83901ed: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.000s I0507 06:30:54.833542 4505 raft_consensus.cc:340] T 143439bb7e50454aab47fae4ed3ad4ae P ae72da07d97b4d9b9e0d246ff83901ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae72da07d97b4d9b9e0d246ff83901ed" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 34185 } } peers { permanent_uuid: "7eecdaa8d1dc4a9988492494fbb17f1d" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 33833 } } peers { permanent_uuid: "a49a0f0a6258477d8126ad3c88bf0d44" member_type: VOTER last_known_addr { host: "127.0.57.196" port: 43761 } } I0507 06:30:54.833542 4504 raft_consensus.cc:340] T 143439bb7e50454aab47fae4ed3ad4ae P 7eecdaa8d1dc4a9988492494fbb17f1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae72da07d97b4d9b9e0d246ff83901ed" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 34185 } } peers { permanent_uuid: "7eecdaa8d1dc4a9988492494fbb17f1d" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 33833 } } peers { permanent_uuid: "a49a0f0a6258477d8126ad3c88bf0d44" member_type: VOTER last_known_addr { host: "127.0.57.196" port: 43761 } } I0507 06:30:54.833883 4504 raft_consensus.cc:366] T 143439bb7e50454aab47fae4ed3ad4ae P 7eecdaa8d1dc4a9988492494fbb17f1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0507 06:30:54.833956 4504 raft_consensus.cc:705] T 143439bb7e50454aab47fae4ed3ad4ae P 7eecdaa8d1dc4a9988492494fbb17f1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7eecdaa8d1dc4a9988492494fbb17f1d, State: Initialized, Role: FOLLOWER I0507 06:30:54.834007 4505 raft_consensus.cc:366] T 143439bb7e50454aab47fae4ed3ad4ae P ae72da07d97b4d9b9e0d246ff83901ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0507 06:30:54.833765 4503 raft_consensus.cc:340] T 143439bb7e50454aab47fae4ed3ad4ae P a49a0f0a6258477d8126ad3c88bf0d44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae72da07d97b4d9b9e0d246ff83901ed" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 34185 } } peers { permanent_uuid: "7eecdaa8d1dc4a9988492494fbb17f1d" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 33833 } } peers { permanent_uuid: "a49a0f0a6258477d8126ad3c88bf0d44" member_type: VOTER last_known_addr { host: "127.0.57.196" port: 43761 } } I0507 06:30:54.834067 4505 raft_consensus.cc:705] T 143439bb7e50454aab47fae4ed3ad4ae P ae72da07d97b4d9b9e0d246ff83901ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae72da07d97b4d9b9e0d246ff83901ed, State: Initialized, Role: FOLLOWER I0507 06:30:54.834174 4503 raft_consensus.cc:366] T 143439bb7e50454aab47fae4ed3ad4ae P a49a0f0a6258477d8126ad3c88bf0d44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0507 06:30:54.834233 4503 raft_consensus.cc:705] T 143439bb7e50454aab47fae4ed3ad4ae P a49a0f0a6258477d8126ad3c88bf0d44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a49a0f0a6258477d8126ad3c88bf0d44, State: Initialized, Role: FOLLOWER I0507 06:30:54.834508 4504 consensus_queue.cc:229] T 143439bb7e50454aab47fae4ed3ad4ae P 7eecdaa8d1dc4a9988492494fbb17f1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae72da07d97b4d9b9e0d246ff83901ed" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 34185 } } peers { permanent_uuid: "7eecdaa8d1dc4a9988492494fbb17f1d" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 33833 } } peers { permanent_uuid: "a49a0f0a6258477d8126ad3c88bf0d44" member_type: VOTER last_known_addr { host: "127.0.57.196" port: 43761 } } I0507 06:30:54.834513 4505 consensus_queue.cc:229] T 143439bb7e50454aab47fae4ed3ad4ae P ae72da07d97b4d9b9e0d246ff83901ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae72da07d97b4d9b9e0d246ff83901ed" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 34185 } } peers { permanent_uuid: "7eecdaa8d1dc4a9988492494fbb17f1d" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 33833 } } peers { permanent_uuid: "a49a0f0a6258477d8126ad3c88bf0d44" member_type: VOTER last_known_addr { host: "127.0.57.196" port: 43761 } } I0507 06:30:54.834658 4503 consensus_queue.cc:229] T 143439bb7e50454aab47fae4ed3ad4ae P a49a0f0a6258477d8126ad3c88bf0d44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae72da07d97b4d9b9e0d246ff83901ed" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 34185 } } peers { permanent_uuid: "7eecdaa8d1dc4a9988492494fbb17f1d" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 33833 } } peers { permanent_uuid: "a49a0f0a6258477d8126ad3c88bf0d44" member_type: VOTER last_known_addr { host: "127.0.57.196" port: 43761 } } I0507 06:30:54.835342 4361 heartbeater.cc:467] Master 127.0.57.254:44959 was elected leader, sending a full tablet report... I0507 06:30:54.835563 4239 heartbeater.cc:467] Master 127.0.57.254:44959 was elected leader, sending a full tablet report... I0507 06:30:54.835733 4483 heartbeater.cc:467] Master 127.0.57.254:44959 was elected leader, sending a full tablet report... I0507 06:30:54.836051 4511 raft_consensus.cc:472] T 143439bb7e50454aab47fae4ed3ad4ae P a49a0f0a6258477d8126ad3c88bf0d44 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0507 06:30:54.836838 4509 raft_consensus.cc:472] T 143439bb7e50454aab47fae4ed3ad4ae P ae72da07d97b4d9b9e0d246ff83901ed [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0507 06:30:54.836925 4503 ts_tablet_manager.cc:1081] T 143439bb7e50454aab47fae4ed3ad4ae P a49a0f0a6258477d8126ad3c88bf0d44: Time spent starting tablet: real 0.009s user 0.007s sys 0.000s I0507 06:30:54.837030 4509 raft_consensus.cc:494] T 143439bb7e50454aab47fae4ed3ad4ae P ae72da07d97b4d9b9e0d246ff83901ed [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae72da07d97b4d9b9e0d246ff83901ed" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 34185 } } peers { permanent_uuid: "7eecdaa8d1dc4a9988492494fbb17f1d" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 33833 } } peers { permanent_uuid: "a49a0f0a6258477d8126ad3c88bf0d44" member_type: VOTER last_known_addr { host: "127.0.57.196" port: 43761 } } I0507 06:30:54.839797 4504 ts_tablet_manager.cc:1081] T 143439bb7e50454aab47fae4ed3ad4ae P 7eecdaa8d1dc4a9988492494fbb17f1d: Time spent starting tablet: real 0.012s user 0.007s sys 0.000s I0507 06:30:54.841058 4505 ts_tablet_manager.cc:1081] T 143439bb7e50454aab47fae4ed3ad4ae P ae72da07d97b4d9b9e0d246ff83901ed: Time spent starting tablet: real 0.013s user 0.007s sys 0.000s I0507 06:30:54.836227 4511 raft_consensus.cc:494] T 143439bb7e50454aab47fae4ed3ad4ae P a49a0f0a6258477d8126ad3c88bf0d44 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae72da07d97b4d9b9e0d246ff83901ed" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 34185 } } peers { permanent_uuid: "7eecdaa8d1dc4a9988492494fbb17f1d" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 33833 } } peers { permanent_uuid: "a49a0f0a6258477d8126ad3c88bf0d44" member_type: VOTER last_known_addr { host: "127.0.57.196" port: 43761 } } I0507 06:30:54.842403 4509 leader_election.cc:296] T 143439bb7e50454aab47fae4ed3ad4ae P ae72da07d97b4d9b9e0d246ff83901ed [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7eecdaa8d1dc4a9988492494fbb17f1d (127.0.57.195:33833), a49a0f0a6258477d8126ad3c88bf0d44 (127.0.57.196:43761) I0507 06:30:54.844599 4511 leader_election.cc:296] T 143439bb7e50454aab47fae4ed3ad4ae P a49a0f0a6258477d8126ad3c88bf0d44 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ae72da07d97b4d9b9e0d246ff83901ed (127.0.57.194:34185), 7eecdaa8d1dc4a9988492494fbb17f1d (127.0.57.195:33833) I0507 06:30:54.847190 4318 tablet_service.cc:1286] Received RequestConsensusVote() RPC: tablet_id: "143439bb7e50454aab47fae4ed3ad4ae" candidate_uuid: "ae72da07d97b4d9b9e0d246ff83901ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7eecdaa8d1dc4a9988492494fbb17f1d" is_pre_election: true I0507 06:30:54.847617 4318 raft_consensus.cc:2346] T 143439bb7e50454aab47fae4ed3ad4ae P 7eecdaa8d1dc4a9988492494fbb17f1d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae72da07d97b4d9b9e0d246ff83901ed in term 0. I0507 06:30:54.848419 4129 leader_election.cc:310] T 143439bb7e50454aab47fae4ed3ad4ae P ae72da07d97b4d9b9e0d246ff83901ed [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7eecdaa8d1dc4a9988492494fbb17f1d, ae72da07d97b4d9b9e0d246ff83901ed; no voters: I0507 06:30:54.849040 4509 raft_consensus.cc:2640] T 143439bb7e50454aab47fae4ed3ad4ae P ae72da07d97b4d9b9e0d246ff83901ed [term 0 FOLLOWER]: Leader pre-election won for term 1 I0507 06:30:54.849193 4509 raft_consensus.cc:472] T 143439bb7e50454aab47fae4ed3ad4ae P ae72da07d97b4d9b9e0d246ff83901ed [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0507 06:30:54.849294 4509 raft_consensus.cc:2886] T 143439bb7e50454aab47fae4ed3ad4ae P ae72da07d97b4d9b9e0d246ff83901ed [term 0 FOLLOWER]: Advancing to term 1 I0507 06:30:54.849205 4196 tablet_service.cc:1286] Received RequestConsensusVote() RPC: tablet_id: "143439bb7e50454aab47fae4ed3ad4ae" candidate_uuid: "a49a0f0a6258477d8126ad3c88bf0d44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae72da07d97b4d9b9e0d246ff83901ed" is_pre_election: true I0507 06:30:54.851205 4440 tablet_service.cc:1286] Received RequestConsensusVote() RPC: tablet_id: "143439bb7e50454aab47fae4ed3ad4ae" candidate_uuid: "ae72da07d97b4d9b9e0d246ff83901ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a49a0f0a6258477d8126ad3c88bf0d44" is_pre_election: true I0507 06:30:54.851474 4440 raft_consensus.cc:2346] T 143439bb7e50454aab47fae4ed3ad4ae P a49a0f0a6258477d8126ad3c88bf0d44 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae72da07d97b4d9b9e0d246ff83901ed in term 0. I0507 06:30:54.852097 4509 raft_consensus.cc:494] T 143439bb7e50454aab47fae4ed3ad4ae P ae72da07d97b4d9b9e0d246ff83901ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae72da07d97b4d9b9e0d246ff83901ed" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 34185 } } peers { permanent_uuid: "7eecdaa8d1dc4a9988492494fbb17f1d" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 33833 } } peers { permanent_uuid: "a49a0f0a6258477d8126ad3c88bf0d44" member_type: VOTER last_known_addr { host: "127.0.57.196" port: 43761 } } I0507 06:30:54.852682 4196 raft_consensus.cc:2276] T 143439bb7e50454aab47fae4ed3ad4ae P ae72da07d97b4d9b9e0d246ff83901ed [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a49a0f0a6258477d8126ad3c88bf0d44 in current term 1: Already voted for candidate ae72da07d97b4d9b9e0d246ff83901ed in this term. I0507 06:30:54.853560 4509 leader_election.cc:296] T 143439bb7e50454aab47fae4ed3ad4ae P ae72da07d97b4d9b9e0d246ff83901ed [CANDIDATE]: Term 1 election: Requested vote from peers 7eecdaa8d1dc4a9988492494fbb17f1d (127.0.57.195:33833), a49a0f0a6258477d8126ad3c88bf0d44 (127.0.57.196:43761) I0507 06:30:54.853937 4440 tablet_service.cc:1286] Received RequestConsensusVote() RPC: tablet_id: "143439bb7e50454aab47fae4ed3ad4ae" candidate_uuid: "ae72da07d97b4d9b9e0d246ff83901ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a49a0f0a6258477d8126ad3c88bf0d44" I0507 06:30:54.854059 4440 raft_consensus.cc:2886] T 143439bb7e50454aab47fae4ed3ad4ae P a49a0f0a6258477d8126ad3c88bf0d44 [term 0 FOLLOWER]: Advancing to term 1 I0507 06:30:54.854548 4318 tablet_service.cc:1286] Received RequestConsensusVote() RPC: tablet_id: "143439bb7e50454aab47fae4ed3ad4ae" candidate_uuid: "ae72da07d97b4d9b9e0d246ff83901ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7eecdaa8d1dc4a9988492494fbb17f1d" I0507 06:30:54.854754 4318 raft_consensus.cc:2886] T 143439bb7e50454aab47fae4ed3ad4ae P 7eecdaa8d1dc4a9988492494fbb17f1d [term 0 FOLLOWER]: Advancing to term 1 I0507 06:30:54.855648 4317 tablet_service.cc:1286] Received RequestConsensusVote() RPC: tablet_id: "143439bb7e50454aab47fae4ed3ad4ae" candidate_uuid: "a49a0f0a6258477d8126ad3c88bf0d44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7eecdaa8d1dc4a9988492494fbb17f1d" is_pre_election: true I0507 06:30:54.856178 4440 raft_consensus.cc:2346] T 143439bb7e50454aab47fae4ed3ad4ae P a49a0f0a6258477d8126ad3c88bf0d44 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae72da07d97b4d9b9e0d246ff83901ed in term 1. I0507 06:30:54.856492 4130 leader_election.cc:310] T 143439bb7e50454aab47fae4ed3ad4ae P ae72da07d97b4d9b9e0d246ff83901ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a49a0f0a6258477d8126ad3c88bf0d44, ae72da07d97b4d9b9e0d246ff83901ed; no voters: I0507 06:30:54.856767 4509 raft_consensus.cc:2640] T 143439bb7e50454aab47fae4ed3ad4ae P ae72da07d97b4d9b9e0d246ff83901ed [term 1 FOLLOWER]: Leader election won for term 1 I0507 06:30:54.857154 4318 raft_consensus.cc:2346] T 143439bb7e50454aab47fae4ed3ad4ae P 7eecdaa8d1dc4a9988492494fbb17f1d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae72da07d97b4d9b9e0d246ff83901ed in term 1. I0507 06:30:54.857273 4509 raft_consensus.cc:667] T 143439bb7e50454aab47fae4ed3ad4ae P ae72da07d97b4d9b9e0d246ff83901ed [term 1 LEADER]: Becoming Leader. State: Replica: ae72da07d97b4d9b9e0d246ff83901ed, State: Running, Role: LEADER I0507 06:30:54.857455 4317 raft_consensus.cc:2321] T 143439bb7e50454aab47fae4ed3ad4ae P 7eecdaa8d1dc4a9988492494fbb17f1d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a49a0f0a6258477d8126ad3c88bf0d44 for term 1 because replica is already servicing an update from a current leader or another vote. I0507 06:30:54.857659 4509 consensus_queue.cc:206] T 143439bb7e50454aab47fae4ed3ad4ae P ae72da07d97b4d9b9e0d246ff83901ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae72da07d97b4d9b9e0d246ff83901ed" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 34185 } } peers { permanent_uuid: "7eecdaa8d1dc4a9988492494fbb17f1d" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 33833 } } peers { permanent_uuid: "a49a0f0a6258477d8126ad3c88bf0d44" member_type: VOTER last_known_addr { host: "127.0.57.196" port: 43761 } } I0507 06:30:54.858263 4373 leader_election.cc:310] T 143439bb7e50454aab47fae4ed3ad4ae P a49a0f0a6258477d8126ad3c88bf0d44 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a49a0f0a6258477d8126ad3c88bf0d44; no voters: 7eecdaa8d1dc4a9988492494fbb17f1d, ae72da07d97b4d9b9e0d246ff83901ed I0507 06:30:54.858672 4511 raft_consensus.cc:2588] T 143439bb7e50454aab47fae4ed3ad4ae P a49a0f0a6258477d8126ad3c88bf0d44 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority I0507 06:30:54.862110 3953 catalog_manager.cc:3988] T 143439bb7e50454aab47fae4ed3ad4ae P ae72da07d97b4d9b9e0d246ff83901ed reported cstate change: term changed from 0 to 1, leader changed from to ae72da07d97b4d9b9e0d246ff83901ed (127.0.57.194). New cstate: current_term: 1 leader_uuid: "ae72da07d97b4d9b9e0d246ff83901ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae72da07d97b4d9b9e0d246ff83901ed" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 34185 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7eecdaa8d1dc4a9988492494fbb17f1d" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 33833 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a49a0f0a6258477d8126ad3c88bf0d44" member_type: VOTER last_known_addr { host: "127.0.57.196" port: 43761 } health_report { overall_health: UNKNOWN } } } W0507 06:30:54.881022 4240 tablet.cc:1786] T 143439bb7e50454aab47fae4ed3ad4ae P ae72da07d97b4d9b9e0d246ff83901ed: Can't schedule compaction. Clean time has not been advanced past its initial value. I0507 06:30:54.884158 231 external_mini_cluster.cc:623] 4 TS(s) registered with all masters I0507 06:30:54.885648 231 ts_itest-base.cc:232] Waiting for 0 tablets on tserver 6f2066e97dec4d75a86e72155048de14 to finish bootstrapping I0507 06:30:54.893504 231 ts_itest-base.cc:232] Waiting for 1 tablets on tserver ae72da07d97b4d9b9e0d246ff83901ed to finish bootstrapping I0507 06:30:54.899319 4116 heartbeater.cc:467] Master 127.0.57.254:44959 was elected leader, sending a full tablet report... I0507 06:30:54.899828 231 ts_itest-base.cc:232] Waiting for 1 tablets on tserver 7eecdaa8d1dc4a9988492494fbb17f1d to finish bootstrapping I0507 06:30:54.899942 4440 raft_consensus.cc:1181] T 143439bb7e50454aab47fae4ed3ad4ae P a49a0f0a6258477d8126ad3c88bf0d44 [term 1 FOLLOWER]: Refusing update from remote peer ae72da07d97b4d9b9e0d246ff83901ed: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch) I0507 06:30:54.900887 4509 consensus_queue.cc:984] T 143439bb7e50454aab47fae4ed3ad4ae P ae72da07d97b4d9b9e0d246ff83901ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "a49a0f0a6258477d8126ad3c88bf0d44" member_type: VOTER last_known_addr { host: "127.0.57.196" port: 43761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0507 06:30:54.903185 4317 raft_consensus.cc:1181] T 143439bb7e50454aab47fae4ed3ad4ae P 7eecdaa8d1dc4a9988492494fbb17f1d [term 1 FOLLOWER]: Refusing update from remote peer ae72da07d97b4d9b9e0d246ff83901ed: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch) I0507 06:30:54.903945 4509 consensus_queue.cc:984] T 143439bb7e50454aab47fae4ed3ad4ae P ae72da07d97b4d9b9e0d246ff83901ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "7eecdaa8d1dc4a9988492494fbb17f1d" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 33833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0507 06:30:54.906386 231 ts_itest-base.cc:232] Waiting for 1 tablets on tserver a49a0f0a6258477d8126ad3c88bf0d44 to finish bootstrapping I0507 06:30:54.934243 4196 consensus_queue.cc:206] T 143439bb7e50454aab47fae4ed3ad4ae P ae72da07d97b4d9b9e0d246ff83901ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae72da07d97b4d9b9e0d246ff83901ed" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 34185 } } peers { permanent_uuid: "7eecdaa8d1dc4a9988492494fbb17f1d" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 33833 } } peers { permanent_uuid: "a49a0f0a6258477d8126ad3c88bf0d44" member_type: VOTER last_known_addr { host: "127.0.57.196" port: 43761 } } peers { permanent_uuid: "6f2066e97dec4d75a86e72155048de14" member_type: NON_VOTER last_known_addr { host: "127.0.57.193" port: 43041 } attrs { } } I0507 06:30:54.936823 4318 raft_consensus.cc:1181] T 143439bb7e50454aab47fae4ed3ad4ae P 7eecdaa8d1dc4a9988492494fbb17f1d [term 1 FOLLOWER]: Refusing update from remote peer ae72da07d97b4d9b9e0d246ff83901ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0507 06:30:54.937038 4440 raft_consensus.cc:1181] T 143439bb7e50454aab47fae4ed3ad4ae P a49a0f0a6258477d8126ad3c88bf0d44 [term 1 FOLLOWER]: Refusing update from remote peer ae72da07d97b4d9b9e0d246ff83901ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0507 06:30:54.937499 4524 consensus_queue.cc:984] T 143439bb7e50454aab47fae4ed3ad4ae P ae72da07d97b4d9b9e0d246ff83901ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "7eecdaa8d1dc4a9988492494fbb17f1d" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 33833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s I0507 06:30:54.937810 4517 consensus_queue.cc:984] T 143439bb7e50454aab47fae4ed3ad4ae P ae72da07d97b4d9b9e0d246ff83901ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "a49a0f0a6258477d8126ad3c88bf0d44" member_type: VOTER last_known_addr { host: "127.0.57.196" port: 43761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s I0507 06:30:54.940167 4517 raft_consensus.cc:2788] T 143439bb7e50454aab47fae4ed3ad4ae P ae72da07d97b4d9b9e0d246ff83901ed [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6f2066e97dec4d75a86e72155048de14 (127.0.57.193) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae72da07d97b4d9b9e0d246ff83901ed" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 34185 } } peers { permanent_uuid: "7eecdaa8d1dc4a9988492494fbb17f1d" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 33833 } } peers { permanent_uuid: "a49a0f0a6258477d8126ad3c88bf0d44" member_type: VOTER last_known_addr { host: "127.0.57.196" port: 43761 } } peers { permanent_uuid: "6f2066e97dec4d75a86e72155048de14" member_type: NON_VOTER last_known_addr { host: "127.0.57.193" port: 43041 } attrs { } } } I0507 06:30:54.940660 4440 raft_consensus.cc:2788] T 143439bb7e50454aab47fae4ed3ad4ae P a49a0f0a6258477d8126ad3c88bf0d44 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6f2066e97dec4d75a86e72155048de14 (127.0.57.193) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae72da07d97b4d9b9e0d246ff83901ed" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 34185 } } peers { permanent_uuid: "7eecdaa8d1dc4a9988492494fbb17f1d" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 33833 } } peers { permanent_uuid: "a49a0f0a6258477d8126ad3c88bf0d44" member_type: VOTER last_known_addr { host: "127.0.57.196" port: 43761 } } peers { permanent_uuid: "6f2066e97dec4d75a86e72155048de14" member_type: NON_VOTER last_known_addr { host: "127.0.57.193" port: 43041 } attrs { } } } I0507 06:30:54.941031 4318 raft_consensus.cc:2788] T 143439bb7e50454aab47fae4ed3ad4ae P 7eecdaa8d1dc4a9988492494fbb17f1d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6f2066e97dec4d75a86e72155048de14 (127.0.57.193) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae72da07d97b4d9b9e0d246ff83901ed" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 34185 } } peers { permanent_uuid: "7eecdaa8d1dc4a9988492494fbb17f1d" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 33833 } } peers { permanent_uuid: "a49a0f0a6258477d8126ad3c88bf0d44" member_type: VOTER last_known_addr { host: "127.0.57.196" port: 43761 } } peers { permanent_uuid: "6f2066e97dec4d75a86e72155048de14" member_type: NON_VOTER last_known_addr { host: "127.0.57.193" port: 43041 } attrs { } } } W0507 06:30:54.942447 4130 consensus_peers.cc:457] T 143439bb7e50454aab47fae4ed3ad4ae P ae72da07d97b4d9b9e0d246ff83901ed -> Peer 6f2066e97dec4d75a86e72155048de14 (127.0.57.193:43041): Couldn't send request to peer 6f2066e97dec4d75a86e72155048de14. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 143439bb7e50454aab47fae4ed3ad4ae. This is attempt 1: this message will repeat every 5th retry. I0507 06:30:54.944687 3951 catalog_manager.cc:3988] T 143439bb7e50454aab47fae4ed3ad4ae P a49a0f0a6258477d8126ad3c88bf0d44 reported cstate change: config changed from index -1 to 2, NON_VOTER 6f2066e97dec4d75a86e72155048de14 (127.0.57.193) added. New cstate: current_term: 1 leader_uuid: "ae72da07d97b4d9b9e0d246ff83901ed" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae72da07d97b4d9b9e0d246ff83901ed" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 34185 } } peers { permanent_uuid: "7eecdaa8d1dc4a9988492494fbb17f1d" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 33833 } } peers { permanent_uuid: "a49a0f0a6258477d8126ad3c88bf0d44" member_type: VOTER last_known_addr { host: "127.0.57.196" port: 43761 } } peers { permanent_uuid: "6f2066e97dec4d75a86e72155048de14" member_type: NON_VOTER last_known_addr { host: "127.0.57.193" port: 43041 } attrs { } } } I0507 06:30:54.999177 4527 ts_tablet_manager.cc:676] T 143439bb7e50454aab47fae4ed3ad4ae P 6f2066e97dec4d75a86e72155048de14: Initiating tablet copy from peer ae72da07d97b4d9b9e0d246ff83901ed (127.0.57.194:34185) I0507 06:30:54.999550 4527 tablet_copy_client.cc:240] T 143439bb7e50454aab47fae4ed3ad4ae P 6f2066e97dec4d75a86e72155048de14: tablet copy: Beginning tablet copy session from remote peer at address 127.0.57.194:34185 I0507 06:30:55.005698 4216 tablet_copy_service.cc:135] P ae72da07d97b4d9b9e0d246ff83901ed: Received BeginTabletCopySession request for tablet 143439bb7e50454aab47fae4ed3ad4ae from peer 6f2066e97dec4d75a86e72155048de14 ({username='slave'} at 127.0.57.193:57783) I0507 06:30:55.005909 4216 tablet_copy_service.cc:156] P ae72da07d97b4d9b9e0d246ff83901ed: Beginning new tablet copy session on tablet 143439bb7e50454aab47fae4ed3ad4ae from peer 6f2066e97dec4d75a86e72155048de14 at {username='slave'} at 127.0.57.193:57783: session id = 6f2066e97dec4d75a86e72155048de14-143439bb7e50454aab47fae4ed3ad4ae I0507 06:30:55.008263 4216 tablet_copy_source_session.cc:202] T 143439bb7e50454aab47fae4ed3ad4ae P ae72da07d97b4d9b9e0d246ff83901ed: Tablet Copy: opened 0 blocks and 1 log segments I0507 06:30:55.013106 4527 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet 143439bb7e50454aab47fae4ed3ad4ae. 1 dirs total, 0 dirs full, 0 dirs failed I0507 06:30:55.022893 4527 tablet_copy_client.cc:575] T 143439bb7e50454aab47fae4ed3ad4ae P 6f2066e97dec4d75a86e72155048de14: tablet copy: Starting download of 0 data blocks... I0507 06:30:55.023075 4527 tablet_copy_client.cc:538] T 143439bb7e50454aab47fae4ed3ad4ae P 6f2066e97dec4d75a86e72155048de14: tablet copy: Starting download of 1 WAL segments... I0507 06:30:55.025172 4527 tablet_copy_client.cc:414] T 143439bb7e50454aab47fae4ed3ad4ae P 6f2066e97dec4d75a86e72155048de14: tablet copy: Tablet Copy complete. Replacing tablet superblock. I0507 06:30:55.028570 4527 tablet_bootstrap.cc:439] T 143439bb7e50454aab47fae4ed3ad4ae P 6f2066e97dec4d75a86e72155048de14: Bootstrap starting. I0507 06:30:55.034040 4527 log.cc:525] T 143439bb7e50454aab47fae4ed3ad4ae P 6f2066e97dec4d75a86e72155048de14: Log is configured to *not* fsync() on all Append() calls I0507 06:30:55.036851 4527 tablet_bootstrap.cc:439] T 143439bb7e50454aab47fae4ed3ad4ae P 6f2066e97dec4d75a86e72155048de14: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I0507 06:30:55.037178 4527 tablet_bootstrap.cc:439] T 143439bb7e50454aab47fae4ed3ad4ae P 6f2066e97dec4d75a86e72155048de14: Bootstrap complete. I0507 06:30:55.037452 4527 ts_tablet_manager.cc:1053] T 143439bb7e50454aab47fae4ed3ad4ae P 6f2066e97dec4d75a86e72155048de14: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.005s I0507 06:30:55.043193 4527 raft_consensus.cc:340] T 143439bb7e50454aab47fae4ed3ad4ae P 6f2066e97dec4d75a86e72155048de14 [term 1 LEARNER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae72da07d97b4d9b9e0d246ff83901ed" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 34185 } } peers { permanent_uuid: "7eecdaa8d1dc4a9988492494fbb17f1d" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 33833 } } peers { permanent_uuid: "a49a0f0a6258477d8126ad3c88bf0d44" member_type: VOTER last_known_addr { host: "127.0.57.196" port: 43761 } } peers { permanent_uuid: "6f2066e97dec4d75a86e72155048de14" member_type: NON_VOTER last_known_addr { host: "127.0.57.193" port: 43041 } attrs { } } I0507 06:30:55.043632 4527 raft_consensus.cc:705] T 143439bb7e50454aab47fae4ed3ad4ae P 6f2066e97dec4d75a86e72155048de14 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6f2066e97dec4d75a86e72155048de14, State: Initialized, Role: LEARNER I0507 06:30:55.044116 4527 consensus_queue.cc:229] T 143439bb7e50454aab47fae4ed3ad4ae P 6f2066e97dec4d75a86e72155048de14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae72da07d97b4d9b9e0d246ff83901ed" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 34185 } } peers { permanent_uuid: "7eecdaa8d1dc4a9988492494fbb17f1d" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 33833 } } peers { permanent_uuid: "a49a0f0a6258477d8126ad3c88bf0d44" member_type: VOTER last_known_addr { host: "127.0.57.196" port: 43761 } } peers { permanent_uuid: "6f2066e97dec4d75a86e72155048de14" member_type: NON_VOTER last_known_addr { host: "127.0.57.193" port: 43041 } attrs { } } I0507 06:30:55.045675 4527 ts_tablet_manager.cc:1081] T 143439bb7e50454aab47fae4ed3ad4ae P 6f2066e97dec4d75a86e72155048de14: Time spent starting tablet: real 0.008s user 0.005s sys 0.004s I0507 06:30:55.046542 4216 tablet_copy_service.cc:325] P ae72da07d97b4d9b9e0d246ff83901ed: Request end of tablet copy session 6f2066e97dec4d75a86e72155048de14-143439bb7e50454aab47fae4ed3ad4ae received from {username='slave'} at 127.0.57.193:57783 I0507 06:30:55.046701 4216 tablet_copy_service.cc:413] P ae72da07d97b4d9b9e0d246ff83901ed: ending tablet copy session 6f2066e97dec4d75a86e72155048de14-143439bb7e50454aab47fae4ed3ad4ae on tablet 143439bb7e50454aab47fae4ed3ad4ae with peer 6f2066e97dec4d75a86e72155048de14 I0507 06:30:55.057103 4073 raft_consensus.cc:1126] T 143439bb7e50454aab47fae4ed3ad4ae P 6f2066e97dec4d75a86e72155048de14 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[] I0507 06:30:55.059342 4196 consensus_queue.cc:206] T 143439bb7e50454aab47fae4ed3ad4ae P ae72da07d97b4d9b9e0d246ff83901ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae72da07d97b4d9b9e0d246ff83901ed" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 34185 } } peers { permanent_uuid: "7eecdaa8d1dc4a9988492494fbb17f1d" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 33833 } } peers { permanent_uuid: "a49a0f0a6258477d8126ad3c88bf0d44" member_type: VOTER last_known_addr { host: "127.0.57.196" port: 43761 } } peers { permanent_uuid: "6f2066e97dec4d75a86e72155048de14" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 43041 } attrs { } } I0507 06:30:55.061334 4073 raft_consensus.cc:1181] T 143439bb7e50454aab47fae4ed3ad4ae P 6f2066e97dec4d75a86e72155048de14 [term 1 LEARNER]: Refusing update from remote peer ae72da07d97b4d9b9e0d246ff83901ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch) I0507 06:30:55.061445 4318 raft_consensus.cc:1181] T 143439bb7e50454aab47fae4ed3ad4ae P 7eecdaa8d1dc4a9988492494fbb17f1d [term 1 FOLLOWER]: Refusing update from remote peer ae72da07d97b4d9b9e0d246ff83901ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch) I0507 06:30:55.061470 4440 raft_consensus.cc:1181] T 143439bb7e50454aab47fae4ed3ad4ae P a49a0f0a6258477d8126ad3c88bf0d44 [term 1 FOLLOWER]: Refusing update from remote peer ae72da07d97b4d9b9e0d246ff83901ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch) I0507 06:30:55.061800 4509 consensus_queue.cc:984] T 143439bb7e50454aab47fae4ed3ad4ae P ae72da07d97b4d9b9e0d246ff83901ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f2066e97dec4d75a86e72155048de14" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 43041 } attrs { }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s I0507 06:30:55.062019 4524 consensus_queue.cc:984] T 143439bb7e50454aab47fae4ed3ad4ae P ae72da07d97b4d9b9e0d246ff83901ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "7eecdaa8d1dc4a9988492494fbb17f1d" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 33833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s I0507 06:30:55.062252 4517 consensus_queue.cc:984] T 143439bb7e50454aab47fae4ed3ad4ae P ae72da07d97b4d9b9e0d246ff83901ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "a49a0f0a6258477d8126ad3c88bf0d44" member_type: VOTER last_known_addr { host: "127.0.57.196" port: 43761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s I0507 06:30:55.065078 4517 raft_consensus.cc:2788] T 143439bb7e50454aab47fae4ed3ad4ae P ae72da07d97b4d9b9e0d246ff83901ed [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6f2066e97dec4d75a86e72155048de14 (127.0.57.193) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae72da07d97b4d9b9e0d246ff83901ed" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 34185 } } peers { permanent_uuid: "7eecdaa8d1dc4a9988492494fbb17f1d" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 33833 } } peers { permanent_uuid: "a49a0f0a6258477d8126ad3c88bf0d44" member_type: VOTER last_known_addr { host: "127.0.57.196" port: 43761 } } peers { permanent_uuid: "6f2066e97dec4d75a86e72155048de14" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 43041 } attrs { } } } I0507 06:30:55.065743 4440 raft_consensus.cc:2788] T 143439bb7e50454aab47fae4ed3ad4ae P a49a0f0a6258477d8126ad3c88bf0d44 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6f2066e97dec4d75a86e72155048de14 (127.0.57.193) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae72da07d97b4d9b9e0d246ff83901ed" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 34185 } } peers { permanent_uuid: "7eecdaa8d1dc4a9988492494fbb17f1d" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 33833 } } peers { permanent_uuid: "a49a0f0a6258477d8126ad3c88bf0d44" member_type: VOTER last_known_addr { host: "127.0.57.196" port: 43761 } } peers { permanent_uuid: "6f2066e97dec4d75a86e72155048de14" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 43041 } attrs { } } } I0507 06:30:55.066874 4073 raft_consensus.cc:2788] T 143439bb7e50454aab47fae4ed3ad4ae P 6f2066e97dec4d75a86e72155048de14 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6f2066e97dec4d75a86e72155048de14 (127.0.57.193) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae72da07d97b4d9b9e0d246ff83901ed" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 34185 } } peers { permanent_uuid: "7eecdaa8d1dc4a9988492494fbb17f1d" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 33833 } } peers { permanent_uuid: "a49a0f0a6258477d8126ad3c88bf0d44" member_type: VOTER last_known_addr { host: "127.0.57.196" port: 43761 } } peers { permanent_uuid: "6f2066e97dec4d75a86e72155048de14" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 43041 } attrs { } } } I0507 06:30:55.068653 4318 raft_consensus.cc:2788] T 143439bb7e50454aab47fae4ed3ad4ae P 7eecdaa8d1dc4a9988492494fbb17f1d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6f2066e97dec4d75a86e72155048de14 (127.0.57.193) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae72da07d97b4d9b9e0d246ff83901ed" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 34185 } } peers { permanent_uuid: "7eecdaa8d1dc4a9988492494fbb17f1d" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 33833 } } peers { permanent_uuid: "a49a0f0a6258477d8126ad3c88bf0d44" member_type: VOTER last_known_addr { host: "127.0.57.196" port: 43761 } } peers { permanent_uuid: "6f2066e97dec4d75a86e72155048de14" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 43041 } attrs { } } } I0507 06:30:55.069774 231 ts_itest-base.cc:232] Waiting for 1 tablets on tserver 6f2066e97dec4d75a86e72155048de14 to finish bootstrapping I0507 06:30:55.069697 3952 catalog_manager.cc:3988] T 143439bb7e50454aab47fae4ed3ad4ae P a49a0f0a6258477d8126ad3c88bf0d44 reported cstate change: config changed from index 2 to 3, 6f2066e97dec4d75a86e72155048de14 (127.0.57.193) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ae72da07d97b4d9b9e0d246ff83901ed" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae72da07d97b4d9b9e0d246ff83901ed" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 34185 } } peers { permanent_uuid: "7eecdaa8d1dc4a9988492494fbb17f1d" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 33833 } } peers { permanent_uuid: "a49a0f0a6258477d8126ad3c88bf0d44" member_type: VOTER last_known_addr { host: "127.0.57.196" port: 43761 } } peers { permanent_uuid: "6f2066e97dec4d75a86e72155048de14" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 43041 } attrs { } } } I0507 06:30:55.071156 231 ts_itest-base.cc:232] Waiting for 1 tablets on tserver ae72da07d97b4d9b9e0d246ff83901ed to finish bootstrapping I0507 06:30:55.073165 231 ts_itest-base.cc:232] Waiting for 1 tablets on tserver 7eecdaa8d1dc4a9988492494fbb17f1d to finish bootstrapping I0507 06:30:55.074617 231 ts_itest-base.cc:232] Waiting for 1 tablets on tserver a49a0f0a6258477d8126ad3c88bf0d44 to finish bootstrapping I0507 06:30:55.216835 4511 raft_consensus.cc:472] T 143439bb7e50454aab47fae4ed3ad4ae P a49a0f0a6258477d8126ad3c88bf0d44 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ae72da07d97b4d9b9e0d246ff83901ed) I0507 06:30:55.217043 4511 raft_consensus.cc:494] T 143439bb7e50454aab47fae4ed3ad4ae P a49a0f0a6258477d8126ad3c88bf0d44 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae72da07d97b4d9b9e0d246ff83901ed" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 34185 } } peers { permanent_uuid: "7eecdaa8d1dc4a9988492494fbb17f1d" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 33833 } } peers { permanent_uuid: "a49a0f0a6258477d8126ad3c88bf0d44" member_type: VOTER last_known_addr { host: "127.0.57.196" port: 43761 } } peers { permanent_uuid: "6f2066e97dec4d75a86e72155048de14" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 43041 } attrs { } } I0507 06:30:55.217761 4511 leader_election.cc:296] T 143439bb7e50454aab47fae4ed3ad4ae P a49a0f0a6258477d8126ad3c88bf0d44 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ae72da07d97b4d9b9e0d246ff83901ed (127.0.57.194:34185), 7eecdaa8d1dc4a9988492494fbb17f1d (127.0.57.195:33833), 6f2066e97dec4d75a86e72155048de14 (127.0.57.193:43041) I0507 06:30:55.218521 4318 tablet_service.cc:1286] Received RequestConsensusVote() RPC: tablet_id: "143439bb7e50454aab47fae4ed3ad4ae" candidate_uuid: "a49a0f0a6258477d8126ad3c88bf0d44" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "7eecdaa8d1dc4a9988492494fbb17f1d" is_pre_election: true I0507 06:30:55.218727 4318 raft_consensus.cc:2346] T 143439bb7e50454aab47fae4ed3ad4ae P 7eecdaa8d1dc4a9988492494fbb17f1d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a49a0f0a6258477d8126ad3c88bf0d44 in term 1. I0507 06:30:55.385656 4510 raft_consensus.cc:472] T 143439bb7e50454aab47fae4ed3ad4ae P 7eecdaa8d1dc4a9988492494fbb17f1d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ae72da07d97b4d9b9e0d246ff83901ed) I0507 06:30:55.385864 4510 raft_consensus.cc:494] T 143439bb7e50454aab47fae4ed3ad4ae P 7eecdaa8d1dc4a9988492494fbb17f1d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae72da07d97b4d9b9e0d246ff83901ed" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 34185 } } peers { permanent_uuid: "7eecdaa8d1dc4a9988492494fbb17f1d" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 33833 } } peers { permanent_uuid: "a49a0f0a6258477d8126ad3c88bf0d44" member_type: VOTER last_known_addr { host: "127.0.57.196" port: 43761 } } peers { permanent_uuid: "6f2066e97dec4d75a86e72155048de14" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 43041 } attrs { } } W0507 06:30:55.386943 4373 leader_election.cc:341] T 143439bb7e50454aab47fae4ed3ad4ae P a49a0f0a6258477d8126ad3c88bf0d44 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ae72da07d97b4d9b9e0d246ff83901ed (127.0.57.194:34185): Timed out: RequestConsensusVote RPC to 127.0.57.194:34185 timed out after 0.169s (SENT) I0507 06:30:55.387022 4511 raft_consensus.cc:472] T 143439bb7e50454aab47fae4ed3ad4ae P a49a0f0a6258477d8126ad3c88bf0d44 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ae72da07d97b4d9b9e0d246ff83901ed) I0507 06:30:55.387145 4510 leader_election.cc:296] T 143439bb7e50454aab47fae4ed3ad4ae P 7eecdaa8d1dc4a9988492494fbb17f1d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ae72da07d97b4d9b9e0d246ff83901ed (127.0.57.194:34185), a49a0f0a6258477d8126ad3c88bf0d44 (127.0.57.196:43761), 6f2066e97dec4d75a86e72155048de14 (127.0.57.193:43041) I0507 06:30:55.387197 4511 raft_consensus.cc:494] T 143439bb7e50454aab47fae4ed3ad4ae P a49a0f0a6258477d8126ad3c88bf0d44 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae72da07d97b4d9b9e0d246ff83901ed" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 34185 } } peers { permanent_uuid: "7eecdaa8d1dc4a9988492494fbb17f1d" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 33833 } } peers { permanent_uuid: "a49a0f0a6258477d8126ad3c88bf0d44" member_type: VOTER last_known_addr { host: "127.0.57.196" port: 43761 } } peers { permanent_uuid: "6f2066e97dec4d75a86e72155048de14" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 43041 } attrs { } } I0507 06:30:55.387907 4511 leader_election.cc:296] T 143439bb7e50454aab47fae4ed3ad4ae P a49a0f0a6258477d8126ad3c88bf0d44 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ae72da07d97b4d9b9e0d246ff83901ed (127.0.57.194:34185), 7eecdaa8d1dc4a9988492494fbb17f1d (127.0.57.195:33833), 6f2066e97dec4d75a86e72155048de14 (127.0.57.193:43041) W0507 06:30:55.388208 4374 leader_election.cc:341] T 143439bb7e50454aab47fae4ed3ad4ae P a49a0f0a6258477d8126ad3c88bf0d44 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6f2066e97dec4d75a86e72155048de14 (127.0.57.193:43041): Timed out: connection negotiation to 127.0.57.193:43041 for RPC RequestConsensusVote timed out after 0.169s (ON_OUTBOUND_QUEUE) I0507 06:30:55.388295 4374 leader_election.cc:310] T 143439bb7e50454aab47fae4ed3ad4ae P a49a0f0a6258477d8126ad3c88bf0d44 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 7eecdaa8d1dc4a9988492494fbb17f1d, a49a0f0a6258477d8126ad3c88bf0d44; no voters: 6f2066e97dec4d75a86e72155048de14, ae72da07d97b4d9b9e0d246ff83901ed I0507 06:30:55.388712 4511 raft_consensus.cc:2588] T 143439bb7e50454aab47fae4ed3ad4ae P a49a0f0a6258477d8126ad3c88bf0d44 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority I0507 06:30:55.389353 4318 tablet_service.cc:1286] Received RequestConsensusVote() RPC: tablet_id: "143439bb7e50454aab47fae4ed3ad4ae" candidate_uuid: "a49a0f0a6258477d8126ad3c88bf0d44" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "7eecdaa8d1dc4a9988492494fbb17f1d" is_pre_election: true I0507 06:30:55.389647 4318 raft_consensus.cc:2346] T 143439bb7e50454aab47fae4ed3ad4ae P 7eecdaa8d1dc4a9988492494fbb17f1d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a49a0f0a6258477d8126ad3c88bf0d44 in term 1. I0507 06:30:55.393311 4440 tablet_service.cc:1286] Received RequestConsensusVote() RPC: tablet_id: "143439bb7e50454aab47fae4ed3ad4ae" candidate_uuid: "7eecdaa8d1dc4a9988492494fbb17f1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "a49a0f0a6258477d8126ad3c88bf0d44" is_pre_election: true I0507 06:30:55.393515 4440 raft_consensus.cc:2346] T 143439bb7e50454aab47fae4ed3ad4ae P a49a0f0a6258477d8126ad3c88bf0d44 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7eecdaa8d1dc4a9988492494fbb17f1d in term 1. W0507 06:30:55.560781 4373 leader_election.cc:341] T 143439bb7e50454aab47fae4ed3ad4ae P a49a0f0a6258477d8126ad3c88bf0d44 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ae72da07d97b4d9b9e0d246ff83901ed (127.0.57.194:34185): Timed out: RequestConsensusVote RPC to 127.0.57.194:34185 timed out after 0.172s (SENT) W0507 06:30:55.561120 4374 leader_election.cc:341] T 143439bb7e50454aab47fae4ed3ad4ae P a49a0f0a6258477d8126ad3c88bf0d44 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6f2066e97dec4d75a86e72155048de14 (127.0.57.193:43041): Timed out: connection negotiation to 127.0.57.193:43041 for RPC RequestConsensusVote timed out after 0.172s (ON_OUTBOUND_QUEUE) I0507 06:30:55.561188 4374 leader_election.cc:310] T 143439bb7e50454aab47fae4ed3ad4ae P a49a0f0a6258477d8126ad3c88bf0d44 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 7eecdaa8d1dc4a9988492494fbb17f1d, a49a0f0a6258477d8126ad3c88bf0d44; no voters: 6f2066e97dec4d75a86e72155048de14, ae72da07d97b4d9b9e0d246ff83901ed I0507 06:30:55.561594 4511 raft_consensus.cc:2588] T 143439bb7e50454aab47fae4ed3ad4ae P a49a0f0a6258477d8126ad3c88bf0d44 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority W0507 06:30:55.568891 4251 leader_election.cc:341] T 143439bb7e50454aab47fae4ed3ad4ae P 7eecdaa8d1dc4a9988492494fbb17f1d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ae72da07d97b4d9b9e0d246ff83901ed (127.0.57.194:34185): Timed out: connection negotiation to 127.0.57.194:34185 for RPC RequestConsensusVote timed out after 0.181s (ON_OUTBOUND_QUEUE) W0507 06:30:55.569327 4252 leader_election.cc:341] T 143439bb7e50454aab47fae4ed3ad4ae P 7eecdaa8d1dc4a9988492494fbb17f1d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6f2066e97dec4d75a86e72155048de14 (127.0.57.193:43041): Timed out: connection negotiation to 127.0.57.193:43041 for RPC RequestConsensusVote timed out after 0.181s (ON_OUTBOUND_QUEUE) I0507 06:30:55.569413 4252 leader_election.cc:310] T 143439bb7e50454aab47fae4ed3ad4ae P 7eecdaa8d1dc4a9988492494fbb17f1d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 7eecdaa8d1dc4a9988492494fbb17f1d, a49a0f0a6258477d8126ad3c88bf0d44; no voters: 6f2066e97dec4d75a86e72155048de14, ae72da07d97b4d9b9e0d246ff83901ed I0507 06:30:55.570056 4510 raft_consensus.cc:2588] T 143439bb7e50454aab47fae4ed3ad4ae P 7eecdaa8d1dc4a9988492494fbb17f1d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority I0507 06:30:55.763964 4510 raft_consensus.cc:472] T 143439bb7e50454aab47fae4ed3ad4ae P 7eecdaa8d1dc4a9988492494fbb17f1d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ae72da07d97b4d9b9e0d246ff83901ed) I0507 06:30:55.764191 4510 raft_consensus.cc:494] T 143439bb7e50454aab47fae4ed3ad4ae P 7eecdaa8d1dc4a9988492494fbb17f1d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae72da07d97b4d9b9e0d246ff83901ed" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 34185 } } peers { permanent_uuid: "7eecdaa8d1dc4a9988492494fbb17f1d" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 33833 } } peers { permanent_uuid: "a49a0f0a6258477d8126ad3c88bf0d44" member_type: VOTER last_known_addr { host: "127.0.57.196" port: 43761 } } peers { permanent_uuid: "6f2066e97dec4d75a86e72155048de14" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 43041 } attrs { } } I0507 06:30:55.765151 4510 leader_election.cc:296] T 143439bb7e50454aab47fae4ed3ad4ae P 7eecdaa8d1dc4a9988492494fbb17f1d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ae72da07d97b4d9b9e0d246ff83901ed (127.0.57.194:34185), a49a0f0a6258477d8126ad3c88bf0d44 (127.0.57.196:43761), 6f2066e97dec4d75a86e72155048de14 (127.0.57.193:43041) I0507 06:30:55.766062 4440 tablet_service.cc:1286] Received RequestConsensusVote() RPC: tablet_id: "143439bb7e50454aab47fae4ed3ad4ae" candidate_uuid: "7eecdaa8d1dc4a9988492494fbb17f1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "a49a0f0a6258477d8126ad3c88bf0d44" is_pre_election: true I0507 06:30:55.766319 4440 raft_consensus.cc:2346] T 143439bb7e50454aab47fae4ed3ad4ae P a49a0f0a6258477d8126ad3c88bf0d44 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7eecdaa8d1dc4a9988492494fbb17f1d in term 1. I0507 06:30:55.953011 4510 raft_consensus.cc:472] T 143439bb7e50454aab47fae4ed3ad4ae P 7eecdaa8d1dc4a9988492494fbb17f1d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ae72da07d97b4d9b9e0d246ff83901ed) I0507 06:30:55.953246 4510 raft_consensus.cc:494] T 143439bb7e50454aab47fae4ed3ad4ae P 7eecdaa8d1dc4a9988492494fbb17f1d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae72da07d97b4d9b9e0d246ff83901ed" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 34185 } } peers { permanent_uuid: "7eecdaa8d1dc4a9988492494fbb17f1d" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 33833 } } peers { permanent_uuid: "a49a0f0a6258477d8126ad3c88bf0d44" member_type: VOTER last_known_addr { host: "127.0.57.196" port: 43761 } } peers { permanent_uuid: "6f2066e97dec4d75a86e72155048de14" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 43041 } attrs { } } W0507 06:30:55.953536 4252 leader_election.cc:341] T 143439bb7e50454aab47fae4ed3ad4ae P 7eecdaa8d1dc4a9988492494fbb17f1d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6f2066e97dec4d75a86e72155048de14 (127.0.57.193:43041): Timed out: connection negotiation to 127.0.57.193:43041 for RPC RequestConsensusVote timed out after 0.188s (ON_OUTBOUND_QUEUE) I0507 06:30:55.954120 4510 leader_election.cc:296] T 143439bb7e50454aab47fae4ed3ad4ae P 7eecdaa8d1dc4a9988492494fbb17f1d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ae72da07d97b4d9b9e0d246ff83901ed (127.0.57.194:34185), a49a0f0a6258477d8126ad3c88bf0d44 (127.0.57.196:43761), 6f2066e97dec4d75a86e72155048de14 (127.0.57.193:43041) W0507 06:30:55.954190 4251 leader_election.cc:341] T 143439bb7e50454aab47fae4ed3ad4ae P 7eecdaa8d1dc4a9988492494fbb17f1d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ae72da07d97b4d9b9e0d246ff83901ed (127.0.57.194:34185): Timed out: connection negotiation to 127.0.57.194:34185 for RPC RequestConsensusVote timed out after 0.188s (ON_OUTBOUND_QUEUE) I0507 06:30:55.954327 4251 leader_election.cc:310] T 143439bb7e50454aab47fae4ed3ad4ae P 7eecdaa8d1dc4a9988492494fbb17f1d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 7eecdaa8d1dc4a9988492494fbb17f1d, a49a0f0a6258477d8126ad3c88bf0d44; no voters: 6f2066e97dec4d75a86e72155048de14, ae72da07d97b4d9b9e0d246ff83901ed I0507 06:30:55.954623 4510 raft_consensus.cc:2588] T 143439bb7e50454aab47fae4ed3ad4ae P 7eecdaa8d1dc4a9988492494fbb17f1d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority I0507 06:30:55.954949 4440 tablet_service.cc:1286] Received RequestConsensusVote() RPC: tablet_id: "143439bb7e50454aab47fae4ed3ad4ae" candidate_uuid: "7eecdaa8d1dc4a9988492494fbb17f1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "a49a0f0a6258477d8126ad3c88bf0d44" is_pre_election: true I0507 06:30:55.955176 4440 raft_consensus.cc:2346] T 143439bb7e50454aab47fae4ed3ad4ae P a49a0f0a6258477d8126ad3c88bf0d44 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7eecdaa8d1dc4a9988492494fbb17f1d in term 1. I0507 06:30:56.155292 4510 raft_consensus.cc:472] T 143439bb7e50454aab47fae4ed3ad4ae P 7eecdaa8d1dc4a9988492494fbb17f1d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ae72da07d97b4d9b9e0d246ff83901ed) I0507 06:30:56.155606 4510 raft_consensus.cc:494] T 143439bb7e50454aab47fae4ed3ad4ae P 7eecdaa8d1dc4a9988492494fbb17f1d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae72da07d97b4d9b9e0d246ff83901ed" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 34185 } } peers { permanent_uuid: "7eecdaa8d1dc4a9988492494fbb17f1d" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 33833 } } peers { permanent_uuid: "a49a0f0a6258477d8126ad3c88bf0d44" member_type: VOTER last_known_addr { host: "127.0.57.196" port: 43761 } } peers { permanent_uuid: "6f2066e97dec4d75a86e72155048de14" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 43041 } attrs { } } I0507 06:30:56.156365 4510 leader_election.cc:296] T 143439bb7e50454aab47fae4ed3ad4ae P 7eecdaa8d1dc4a9988492494fbb17f1d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ae72da07d97b4d9b9e0d246ff83901ed (127.0.57.194:34185), a49a0f0a6258477d8126ad3c88bf0d44 (127.0.57.196:43761), 6f2066e97dec4d75a86e72155048de14 (127.0.57.193:43041) I0507 06:30:56.157236 4440 tablet_service.cc:1286] Received RequestConsensusVote() RPC: tablet_id: "143439bb7e50454aab47fae4ed3ad4ae" candidate_uuid: "7eecdaa8d1dc4a9988492494fbb17f1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "a49a0f0a6258477d8126ad3c88bf0d44" is_pre_election: true I0507 06:30:56.157440 4440 raft_consensus.cc:2346] T 143439bb7e50454aab47fae4ed3ad4ae P a49a0f0a6258477d8126ad3c88bf0d44 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7eecdaa8d1dc4a9988492494fbb17f1d in term 1. W0507 06:30:56.238245 4251 leader_election.cc:341] T 143439bb7e50454aab47fae4ed3ad4ae P 7eecdaa8d1dc4a9988492494fbb17f1d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ae72da07d97b4d9b9e0d246ff83901ed (127.0.57.194:34185): Timed out: connection negotiation to 127.0.57.194:34185 for RPC RequestConsensusVote timed out after 0.283s (ON_OUTBOUND_QUEUE) W0507 06:30:56.238433 4252 leader_election.cc:341] T 143439bb7e50454aab47fae4ed3ad4ae P 7eecdaa8d1dc4a9988492494fbb17f1d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6f2066e97dec4d75a86e72155048de14 (127.0.57.193:43041): Timed out: connection negotiation to 127.0.57.193:43041 for RPC RequestConsensusVote timed out after 0.283s (ON_OUTBOUND_QUEUE) I0507 06:30:56.238556 4252 leader_election.cc:310] T 143439bb7e50454aab47fae4ed3ad4ae P 7eecdaa8d1dc4a9988492494fbb17f1d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 7eecdaa8d1dc4a9988492494fbb17f1d, a49a0f0a6258477d8126ad3c88bf0d44; no voters: 6f2066e97dec4d75a86e72155048de14, ae72da07d97b4d9b9e0d246ff83901ed I0507 06:30:56.239018 4510 raft_consensus.cc:2588] T 143439bb7e50454aab47fae4ed3ad4ae P 7eecdaa8d1dc4a9988492494fbb17f1d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority I0507 06:30:56.328275 4536 raft_consensus.cc:472] T 143439bb7e50454aab47fae4ed3ad4ae P a49a0f0a6258477d8126ad3c88bf0d44 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ae72da07d97b4d9b9e0d246ff83901ed) I0507 06:30:56.328433 4536 raft_consensus.cc:494] T 143439bb7e50454aab47fae4ed3ad4ae P a49a0f0a6258477d8126ad3c88bf0d44 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae72da07d97b4d9b9e0d246ff83901ed" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 34185 } } peers { permanent_uuid: "7eecdaa8d1dc4a9988492494fbb17f1d" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 33833 } } peers { permanent_uuid: "a49a0f0a6258477d8126ad3c88bf0d44" member_type: VOTER last_known_addr { host: "127.0.57.196" port: 43761 } } peers { permanent_uuid: "6f2066e97dec4d75a86e72155048de14" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 43041 } attrs { } } I0507 06:30:56.329175 4536 leader_election.cc:296] T 143439bb7e50454aab47fae4ed3ad4ae P a49a0f0a6258477d8126ad3c88bf0d44 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ae72da07d97b4d9b9e0d246ff83901ed (127.0.57.194:34185), 7eecdaa8d1dc4a9988492494fbb17f1d (127.0.57.195:33833), 6f2066e97dec4d75a86e72155048de14 (127.0.57.193:43041) I0507 06:30:56.329849 4318 tablet_service.cc:1286] Received RequestConsensusVote() RPC: tablet_id: "143439bb7e50454aab47fae4ed3ad4ae" candidate_uuid: "a49a0f0a6258477d8126ad3c88bf0d44" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "7eecdaa8d1dc4a9988492494fbb17f1d" is_pre_election: true I0507 06:30:56.330068 4318 raft_consensus.cc:2346] T 143439bb7e50454aab47fae4ed3ad4ae P 7eecdaa8d1dc4a9988492494fbb17f1d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a49a0f0a6258477d8126ad3c88bf0d44 in term 1. W0507 06:30:56.412803 4252 leader_election.cc:341] T 143439bb7e50454aab47fae4ed3ad4ae P 7eecdaa8d1dc4a9988492494fbb17f1d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6f2066e97dec4d75a86e72155048de14 (127.0.57.193:43041): Timed out: connection negotiation to 127.0.57.193:43041 for RPC RequestConsensusVote timed out after 0.256s (ON_OUTBOUND_QUEUE) W0507 06:30:56.413008 4251 leader_election.cc:341] T 143439bb7e50454aab47fae4ed3ad4ae P 7eecdaa8d1dc4a9988492494fbb17f1d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ae72da07d97b4d9b9e0d246ff83901ed (127.0.57.194:34185): Timed out: connection negotiation to 127.0.57.194:34185 for RPC RequestConsensusVote timed out after 0.256s (ON_OUTBOUND_QUEUE) I0507 06:30:56.413120 4251 leader_election.cc:310] T 143439bb7e50454aab47fae4ed3ad4ae P 7eecdaa8d1dc4a9988492494fbb17f1d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 7eecdaa8d1dc4a9988492494fbb17f1d, a49a0f0a6258477d8126ad3c88bf0d44; no voters: 6f2066e97dec4d75a86e72155048de14, ae72da07d97b4d9b9e0d246ff83901ed I0507 06:30:56.413496 4510 raft_consensus.cc:2588] T 143439bb7e50454aab47fae4ed3ad4ae P 7eecdaa8d1dc4a9988492494fbb17f1d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority I0507 06:30:56.601639 4510 raft_consensus.cc:472] T 143439bb7e50454aab47fae4ed3ad4ae P 7eecdaa8d1dc4a9988492494fbb17f1d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ae72da07d97b4d9b9e0d246ff83901ed) I0507 06:30:56.601855 4510 raft_consensus.cc:494] T 143439bb7e50454aab47fae4ed3ad4ae P 7eecdaa8d1dc4a9988492494fbb17f1d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae72da07d97b4d9b9e0d246ff83901ed" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 34185 } } peers { permanent_uuid: "7eecdaa8d1dc4a9988492494fbb17f1d" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 33833 } } peers { permanent_uuid: "a49a0f0a6258477d8126ad3c88bf0d44" member_type: VOTER last_known_addr { host: "127.0.57.196" port: 43761 } } peers { permanent_uuid: "6f2066e97dec4d75a86e72155048de14" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 43041 } attrs { } } I0507 06:30:56.602609 4510 leader_election.cc:296] T 143439bb7e50454aab47fae4ed3ad4ae P 7eecdaa8d1dc4a9988492494fbb17f1d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ae72da07d97b4d9b9e0d246ff83901ed (127.0.57.194:34185), a49a0f0a6258477d8126ad3c88bf0d44 (127.0.57.196:43761), 6f2066e97dec4d75a86e72155048de14 (127.0.57.193:43041) I0507 06:30:56.603415 4440 tablet_service.cc:1286] Received RequestConsensusVote() RPC: tablet_id: "143439bb7e50454aab47fae4ed3ad4ae" candidate_uuid: "7eecdaa8d1dc4a9988492494fbb17f1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "a49a0f0a6258477d8126ad3c88bf0d44" is_pre_election: true I0507 06:30:56.603605 4440 raft_consensus.cc:2346] T 143439bb7e50454aab47fae4ed3ad4ae P a49a0f0a6258477d8126ad3c88bf0d44 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7eecdaa8d1dc4a9988492494fbb17f1d in term 1. W0507 06:30:56.655591 4374 leader_election.cc:341] T 143439bb7e50454aab47fae4ed3ad4ae P a49a0f0a6258477d8126ad3c88bf0d44 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6f2066e97dec4d75a86e72155048de14 (127.0.57.193:43041): Timed out: connection negotiation to 127.0.57.193:43041 for RPC RequestConsensusVote timed out after 0.326s (ON_OUTBOUND_QUEUE) W0507 06:30:56.655817 4373 leader_election.cc:341] T 143439bb7e50454aab47fae4ed3ad4ae P a49a0f0a6258477d8126ad3c88bf0d44 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ae72da07d97b4d9b9e0d246ff83901ed (127.0.57.194:34185): Timed out: RequestConsensusVote RPC to 127.0.57.194:34185 timed out after 0.326s (SENT) I0507 06:30:56.655969 4373 leader_election.cc:310] T 143439bb7e50454aab47fae4ed3ad4ae P a49a0f0a6258477d8126ad3c88bf0d44 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 7eecdaa8d1dc4a9988492494fbb17f1d, a49a0f0a6258477d8126ad3c88bf0d44; no voters: 6f2066e97dec4d75a86e72155048de14, ae72da07d97b4d9b9e0d246ff83901ed I0507 06:30:56.656340 4536 raft_consensus.cc:2588] T 143439bb7e50454aab47fae4ed3ad4ae P a49a0f0a6258477d8126ad3c88bf0d44 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority I0507 06:30:56.934231 4536 raft_consensus.cc:472] T 143439bb7e50454aab47fae4ed3ad4ae P a49a0f0a6258477d8126ad3c88bf0d44 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ae72da07d97b4d9b9e0d246ff83901ed) I0507 06:30:56.934420 4536 raft_consensus.cc:494] T 143439bb7e50454aab47fae4ed3ad4ae P a49a0f0a6258477d8126ad3c88bf0d44 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae72da07d97b4d9b9e0d246ff83901ed" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 34185 } } peers { permanent_uuid: "7eecdaa8d1dc4a9988492494fbb17f1d" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 33833 } } peers { permanent_uuid: "a49a0f0a6258477d8126ad3c88bf0d44" member_type: VOTER last_known_addr { host: "127.0.57.196" port: 43761 } } peers { permanent_uuid: "6f2066e97dec4d75a86e72155048de14" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 43041 } attrs { } } I0507 06:30:56.935236 4536 leader_election.cc:296] T 143439bb7e50454aab47fae4ed3ad4ae P a49a0f0a6258477d8126ad3c88bf0d44 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ae72da07d97b4d9b9e0d246ff83901ed (127.0.57.194:34185), 7eecdaa8d1dc4a9988492494fbb17f1d (127.0.57.195:33833), 6f2066e97dec4d75a86e72155048de14 (127.0.57.193:43041) I0507 06:30:56.935945 4318 tablet_service.cc:1286] Received RequestConsensusVote() RPC: tablet_id: "143439bb7e50454aab47fae4ed3ad4ae" candidate_uuid: "a49a0f0a6258477d8126ad3c88bf0d44" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "7eecdaa8d1dc4a9988492494fbb17f1d" is_pre_election: true I0507 06:30:56.936178 4318 raft_consensus.cc:2346] T 143439bb7e50454aab47fae4ed3ad4ae P 7eecdaa8d1dc4a9988492494fbb17f1d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a49a0f0a6258477d8126ad3c88bf0d44 in term 1. I0507 06:30:57.184130 4536 raft_consensus.cc:472] T 143439bb7e50454aab47fae4ed3ad4ae P a49a0f0a6258477d8126ad3c88bf0d44 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ae72da07d97b4d9b9e0d246ff83901ed) W0507 06:30:57.184470 4374 leader_election.cc:341] T 143439bb7e50454aab47fae4ed3ad4ae P a49a0f0a6258477d8126ad3c88bf0d44 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6f2066e97dec4d75a86e72155048de14 (127.0.57.193:43041): Timed out: connection negotiation to 127.0.57.193:43041 for RPC RequestConsensusVote timed out after 0.249s (ON_OUTBOUND_QUEUE) I0507 06:30:57.184357 4536 raft_consensus.cc:494] T 143439bb7e50454aab47fae4ed3ad4ae P a49a0f0a6258477d8126ad3c88bf0d44 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae72da07d97b4d9b9e0d246ff83901ed" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 34185 } } peers { permanent_uuid: "7eecdaa8d1dc4a9988492494fbb17f1d" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 33833 } } peers { permanent_uuid: "a49a0f0a6258477d8126ad3c88bf0d44" member_type: VOTER last_known_addr { host: "127.0.57.196" port: 43761 } } peers { permanent_uuid: "6f2066e97dec4d75a86e72155048de14" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 43041 } attrs { } } W0507 06:30:57.184675 4373 leader_election.cc:341] T 143439bb7e50454aab47fae4ed3ad4ae P a49a0f0a6258477d8126ad3c88bf0d44 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ae72da07d97b4d9b9e0d246ff83901ed (127.0.57.194:34185): Timed out: RequestConsensusVote RPC to 127.0.57.194:34185 timed out after 0.249s (SENT) I0507 06:30:57.184800 4373 leader_election.cc:310] T 143439bb7e50454aab47fae4ed3ad4ae P a49a0f0a6258477d8126ad3c88bf0d44 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 7eecdaa8d1dc4a9988492494fbb17f1d, a49a0f0a6258477d8126ad3c88bf0d44; no voters: 6f2066e97dec4d75a86e72155048de14, ae72da07d97b4d9b9e0d246ff83901ed I0507 06:30:57.185180 4536 leader_election.cc:296] T 143439bb7e50454aab47fae4ed3ad4ae P a49a0f0a6258477d8126ad3c88bf0d44 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ae72da07d97b4d9b9e0d246ff83901ed (127.0.57.194:34185), 7eecdaa8d1dc4a9988492494fbb17f1d (127.0.57.195:33833), 6f2066e97dec4d75a86e72155048de14 (127.0.57.193:43041) I0507 06:30:57.185407 4536 raft_consensus.cc:2588] T 143439bb7e50454aab47fae4ed3ad4ae P a49a0f0a6258477d8126ad3c88bf0d44 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority I0507 06:30:57.186480 4318 tablet_service.cc:1286] Received RequestConsensusVote() RPC: tablet_id: "143439bb7e50454aab47fae4ed3ad4ae" candidate_uuid: "a49a0f0a6258477d8126ad3c88bf0d44" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "7eecdaa8d1dc4a9988492494fbb17f1d" is_pre_election: true I0507 06:30:57.186827 4318 raft_consensus.cc:2346] T 143439bb7e50454aab47fae4ed3ad4ae P 7eecdaa8d1dc4a9988492494fbb17f1d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a49a0f0a6258477d8126ad3c88bf0d44 in term 1. W0507 06:30:57.411134 4374 leader_election.cc:341] T 143439bb7e50454aab47fae4ed3ad4ae P a49a0f0a6258477d8126ad3c88bf0d44 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6f2066e97dec4d75a86e72155048de14 (127.0.57.193:43041): Timed out: connection negotiation to 127.0.57.193:43041 for RPC RequestConsensusVote timed out after 0.225s (ON_OUTBOUND_QUEUE) W0507 06:30:57.411361 4373 leader_election.cc:341] T 143439bb7e50454aab47fae4ed3ad4ae P a49a0f0a6258477d8126ad3c88bf0d44 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ae72da07d97b4d9b9e0d246ff83901ed (127.0.57.194:34185): Timed out: RequestConsensusVote RPC to 127.0.57.194:34185 timed out after 0.225s (SENT) I0507 06:30:57.411499 4373 leader_election.cc:310] T 143439bb7e50454aab47fae4ed3ad4ae P a49a0f0a6258477d8126ad3c88bf0d44 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 7eecdaa8d1dc4a9988492494fbb17f1d, a49a0f0a6258477d8126ad3c88bf0d44; no voters: 6f2066e97dec4d75a86e72155048de14, ae72da07d97b4d9b9e0d246ff83901ed I0507 06:30:57.411957 4537 raft_consensus.cc:2588] T 143439bb7e50454aab47fae4ed3ad4ae P a49a0f0a6258477d8126ad3c88bf0d44 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority W0507 06:30:57.595466 4251 leader_election.cc:341] T 143439bb7e50454aab47fae4ed3ad4ae P 7eecdaa8d1dc4a9988492494fbb17f1d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ae72da07d97b4d9b9e0d246ff83901ed (127.0.57.194:34185): Timed out: connection negotiation to 127.0.57.194:34185 for RPC RequestConsensusVote timed out after 0.992s (ON_OUTBOUND_QUEUE) W0507 06:30:57.595707 4252 leader_election.cc:341] T 143439bb7e50454aab47fae4ed3ad4ae P 7eecdaa8d1dc4a9988492494fbb17f1d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6f2066e97dec4d75a86e72155048de14 (127.0.57.193:43041): Timed out: connection negotiation to 127.0.57.193:43041 for RPC RequestConsensusVote timed out after 0.992s (ON_OUTBOUND_QUEUE) I0507 06:30:57.595829 4252 leader_election.cc:310] T 143439bb7e50454aab47fae4ed3ad4ae P 7eecdaa8d1dc4a9988492494fbb17f1d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 7eecdaa8d1dc4a9988492494fbb17f1d, a49a0f0a6258477d8126ad3c88bf0d44; no voters: 6f2066e97dec4d75a86e72155048de14, ae72da07d97b4d9b9e0d246ff83901ed I0507 06:30:57.596741 4538 raft_consensus.cc:2588] T 143439bb7e50454aab47fae4ed3ad4ae P 7eecdaa8d1dc4a9988492494fbb17f1d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority I0507 06:30:58.261314 4539 raft_consensus.cc:472] T 143439bb7e50454aab47fae4ed3ad4ae P 7eecdaa8d1dc4a9988492494fbb17f1d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ae72da07d97b4d9b9e0d246ff83901ed) I0507 06:30:58.261488 4539 raft_consensus.cc:494] T 143439bb7e50454aab47fae4ed3ad4ae P 7eecdaa8d1dc4a9988492494fbb17f1d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae72da07d97b4d9b9e0d246ff83901ed" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 34185 } } peers { permanent_uuid: "7eecdaa8d1dc4a9988492494fbb17f1d" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 33833 } } peers { permanent_uuid: "a49a0f0a6258477d8126ad3c88bf0d44" member_type: VOTER last_known_addr { host: "127.0.57.196" port: 43761 } } peers { permanent_uuid: "6f2066e97dec4d75a86e72155048de14" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 43041 } attrs { } } I0507 06:30:58.262187 4539 leader_election.cc:296] T 143439bb7e50454aab47fae4ed3ad4ae P 7eecdaa8d1dc4a9988492494fbb17f1d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ae72da07d97b4d9b9e0d246ff83901ed (127.0.57.194:34185), a49a0f0a6258477d8126ad3c88bf0d44 (127.0.57.196:43761), 6f2066e97dec4d75a86e72155048de14 (127.0.57.193:43041) I0507 06:30:58.262924 4440 tablet_service.cc:1286] Received RequestConsensusVote() RPC: tablet_id: "143439bb7e50454aab47fae4ed3ad4ae" candidate_uuid: "7eecdaa8d1dc4a9988492494fbb17f1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "a49a0f0a6258477d8126ad3c88bf0d44" is_pre_election: true I0507 06:30:58.263164 4440 raft_consensus.cc:2346] T 143439bb7e50454aab47fae4ed3ad4ae P a49a0f0a6258477d8126ad3c88bf0d44 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7eecdaa8d1dc4a9988492494fbb17f1d in term 1. W0507 06:30:58.433686 4515 negotiation.cc:313] Failed RPC negotiation. Trace: 0507 06:30:55.218080 (+ 0us) reactor.cc:583] Submitting negotiation task for client connection to 127.0.57.193:43041 0507 06:30:55.218315 (+ 235us) negotiation.cc:98] Waiting for socket to connect 0507 06:30:55.218332 (+ 17us) client_negotiation.cc:167] Beginning negotiation 0507 06:30:55.218442 (+ 110us) client_negotiation.cc:244] Sending NEGOTIATE NegotiatePB request 0507 06:30:58.432857 (+3214415us) negotiation.cc:304] Negotiation complete: Timed out: Client connection negotiation failed: client connection to 127.0.57.193:43041 Metrics: {"client-negotiator.queue_time_us":85} W0507 06:30:58.433800 4533 negotiation.cc:313] Failed RPC negotiation. Trace: 0507 06:30:55.388384 (+ 0us) reactor.cc:583] Submitting negotiation task for client connection to 127.0.57.193:43041 0507 06:30:55.392407 (+ 4023us) negotiation.cc:98] Waiting for socket to connect 0507 06:30:55.392419 (+ 12us) client_negotiation.cc:167] Beginning negotiation 0507 06:30:55.392496 (+ 77us) client_negotiation.cc:244] Sending NEGOTIATE NegotiatePB request 0507 06:30:58.432865 (+3040369us) negotiation.cc:304] Negotiation complete: Timed out: Client connection negotiation failed: client connection to 127.0.57.193:43041 Metrics: {"client-negotiator.queue_time_us":3936,"thread_start_us":4341,"threads_started":1} W0507 06:30:58.434361 4252 leader_election.cc:341] T 143439bb7e50454aab47fae4ed3ad4ae P 7eecdaa8d1dc4a9988492494fbb17f1d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6f2066e97dec4d75a86e72155048de14 (127.0.57.193:43041): Timed out: Client connection negotiation failed: client connection to 127.0.57.193:43041 W0507 06:30:58.434444 4534 negotiation.cc:313] Failed RPC negotiation. Trace: 0507 06:30:55.387577 (+ 0us) reactor.cc:583] Submitting negotiation task for client connection to 127.0.57.194:34185 0507 06:30:55.389132 (+ 1555us) negotiation.cc:98] Waiting for socket to connect 0507 06:30:55.389149 (+ 17us) client_negotiation.cc:167] Beginning negotiation 0507 06:30:55.389272 (+ 123us) client_negotiation.cc:244] Sending NEGOTIATE NegotiatePB request 0507 06:30:58.432954 (+3043682us) negotiation.cc:304] Negotiation complete: Timed out: Client connection negotiation failed: client connection to 127.0.57.194:34185 Metrics: {"client-negotiator.queue_time_us":1393,"thread_start_us":1253,"threads_started":1} W0507 06:30:58.434612 4251 leader_election.cc:341] T 143439bb7e50454aab47fae4ed3ad4ae P 7eecdaa8d1dc4a9988492494fbb17f1d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ae72da07d97b4d9b9e0d246ff83901ed (127.0.57.194:34185): Timed out: Client connection negotiation failed: client connection to 127.0.57.194:34185 I0507 06:30:58.434720 4251 leader_election.cc:310] T 143439bb7e50454aab47fae4ed3ad4ae P 7eecdaa8d1dc4a9988492494fbb17f1d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 7eecdaa8d1dc4a9988492494fbb17f1d, a49a0f0a6258477d8126ad3c88bf0d44; no voters: 6f2066e97dec4d75a86e72155048de14, ae72da07d97b4d9b9e0d246ff83901ed I0507 06:30:58.435051 4539 raft_consensus.cc:2588] T 143439bb7e50454aab47fae4ed3ad4ae P 7eecdaa8d1dc4a9988492494fbb17f1d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority I0507 06:30:58.567137 4540 raft_consensus.cc:472] T 143439bb7e50454aab47fae4ed3ad4ae P a49a0f0a6258477d8126ad3c88bf0d44 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ae72da07d97b4d9b9e0d246ff83901ed) I0507 06:30:58.567302 4540 raft_consensus.cc:494] T 143439bb7e50454aab47fae4ed3ad4ae P a49a0f0a6258477d8126ad3c88bf0d44 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae72da07d97b4d9b9e0d246ff83901ed" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 34185 } } peers { permanent_uuid: "7eecdaa8d1dc4a9988492494fbb17f1d" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 33833 } } peers { permanent_uuid: "a49a0f0a6258477d8126ad3c88bf0d44" member_type: VOTER last_known_addr { host: "127.0.57.196" port: 43761 } } peers { permanent_uuid: "6f2066e97dec4d75a86e72155048de14" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 43041 } attrs { } } I0507 06:30:58.568061 4540 leader_election.cc:296] T 143439bb7e50454aab47fae4ed3ad4ae P a49a0f0a6258477d8126ad3c88bf0d44 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ae72da07d97b4d9b9e0d246ff83901ed (127.0.57.194:34185), 7eecdaa8d1dc4a9988492494fbb17f1d (127.0.57.195:33833), 6f2066e97dec4d75a86e72155048de14 (127.0.57.193:43041) I0507 06:30:58.568948 4318 tablet_service.cc:1286] Received RequestConsensusVote() RPC: tablet_id: "143439bb7e50454aab47fae4ed3ad4ae" candidate_uuid: "a49a0f0a6258477d8126ad3c88bf0d44" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "7eecdaa8d1dc4a9988492494fbb17f1d" is_pre_election: true I0507 06:30:58.569214 4318 raft_consensus.cc:2346] T 143439bb7e50454aab47fae4ed3ad4ae P 7eecdaa8d1dc4a9988492494fbb17f1d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a49a0f0a6258477d8126ad3c88bf0d44 in term 1. I0507 06:30:59.779325 4541 raft_consensus.cc:472] T 143439bb7e50454aab47fae4ed3ad4ae P a49a0f0a6258477d8126ad3c88bf0d44 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ae72da07d97b4d9b9e0d246ff83901ed) I0507 06:30:59.779498 4541 raft_consensus.cc:494] T 143439bb7e50454aab47fae4ed3ad4ae P a49a0f0a6258477d8126ad3c88bf0d44 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae72da07d97b4d9b9e0d246ff83901ed" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 34185 } } peers { permanent_uuid: "7eecdaa8d1dc4a9988492494fbb17f1d" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 33833 } } peers { permanent_uuid: "a49a0f0a6258477d8126ad3c88bf0d44" member_type: VOTER last_known_addr { host: "127.0.57.196" port: 43761 } } peers { permanent_uuid: "6f2066e97dec4d75a86e72155048de14" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 43041 } attrs { } } I0507 06:30:59.780189 4541 leader_election.cc:296] T 143439bb7e50454aab47fae4ed3ad4ae P a49a0f0a6258477d8126ad3c88bf0d44 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ae72da07d97b4d9b9e0d246ff83901ed (127.0.57.194:34185), 7eecdaa8d1dc4a9988492494fbb17f1d (127.0.57.195:33833), 6f2066e97dec4d75a86e72155048de14 (127.0.57.193:43041) W0507 06:30:59.780205 4374 leader_election.cc:341] T 143439bb7e50454aab47fae4ed3ad4ae P a49a0f0a6258477d8126ad3c88bf0d44 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6f2066e97dec4d75a86e72155048de14 (127.0.57.193:43041): Timed out: connection negotiation to 127.0.57.193:43041 for RPC RequestConsensusVote timed out after 1.211s (ON_OUTBOUND_QUEUE) W0507 06:30:59.780330 4373 leader_election.cc:341] T 143439bb7e50454aab47fae4ed3ad4ae P a49a0f0a6258477d8126ad3c88bf0d44 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ae72da07d97b4d9b9e0d246ff83901ed (127.0.57.194:34185): Timed out: RequestConsensusVote RPC to 127.0.57.194:34185 timed out after 1.211s (SENT) I0507 06:30:59.780704 4373 leader_election.cc:310] T 143439bb7e50454aab47fae4ed3ad4ae P a49a0f0a6258477d8126ad3c88bf0d44 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 7eecdaa8d1dc4a9988492494fbb17f1d, a49a0f0a6258477d8126ad3c88bf0d44; no voters: 6f2066e97dec4d75a86e72155048de14, ae72da07d97b4d9b9e0d246ff83901ed I0507 06:30:59.780956 4541 raft_consensus.cc:2588] T 143439bb7e50454aab47fae4ed3ad4ae P a49a0f0a6258477d8126ad3c88bf0d44 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority I0507 06:30:59.781841 4318 tablet_service.cc:1286] Received RequestConsensusVote() RPC: tablet_id: "143439bb7e50454aab47fae4ed3ad4ae" candidate_uuid: "a49a0f0a6258477d8126ad3c88bf0d44" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "7eecdaa8d1dc4a9988492494fbb17f1d" is_pre_election: true I0507 06:30:59.782049 4318 raft_consensus.cc:2346] T 143439bb7e50454aab47fae4ed3ad4ae P 7eecdaa8d1dc4a9988492494fbb17f1d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a49a0f0a6258477d8126ad3c88bf0d44 in term 1. I0507 06:31:00.346777 4542 raft_consensus.cc:472] T 143439bb7e50454aab47fae4ed3ad4ae P a49a0f0a6258477d8126ad3c88bf0d44 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ae72da07d97b4d9b9e0d246ff83901ed) I0507 06:31:00.346920 4542 raft_consensus.cc:494] T 143439bb7e50454aab47fae4ed3ad4ae P a49a0f0a6258477d8126ad3c88bf0d44 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae72da07d97b4d9b9e0d246ff83901ed" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 34185 } } peers { permanent_uuid: "7eecdaa8d1dc4a9988492494fbb17f1d" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 33833 } } peers { permanent_uuid: "a49a0f0a6258477d8126ad3c88bf0d44" member_type: VOTER last_known_addr { host: "127.0.57.196" port: 43761 } } peers { permanent_uuid: "6f2066e97dec4d75a86e72155048de14" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 43041 } attrs { } } I0507 06:31:00.347641 4542 leader_election.cc:296] T 143439bb7e50454aab47fae4ed3ad4ae P a49a0f0a6258477d8126ad3c88bf0d44 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ae72da07d97b4d9b9e0d246ff83901ed (127.0.57.194:34185), 7eecdaa8d1dc4a9988492494fbb17f1d (127.0.57.195:33833), 6f2066e97dec4d75a86e72155048de14 (127.0.57.193:43041) I0507 06:31:00.348351 4318 tablet_service.cc:1286] Received RequestConsensusVote() RPC: tablet_id: "143439bb7e50454aab47fae4ed3ad4ae" candidate_uuid: "a49a0f0a6258477d8126ad3c88bf0d44" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "7eecdaa8d1dc4a9988492494fbb17f1d" is_pre_election: true I0507 06:31:00.348587 4318 raft_consensus.cc:2346] T 143439bb7e50454aab47fae4ed3ad4ae P 7eecdaa8d1dc4a9988492494fbb17f1d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a49a0f0a6258477d8126ad3c88bf0d44 in term 1. I0507 06:31:00.627832 4543 raft_consensus.cc:472] T 143439bb7e50454aab47fae4ed3ad4ae P 7eecdaa8d1dc4a9988492494fbb17f1d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ae72da07d97b4d9b9e0d246ff83901ed) I0507 06:31:00.627992 4543 raft_consensus.cc:494] T 143439bb7e50454aab47fae4ed3ad4ae P 7eecdaa8d1dc4a9988492494fbb17f1d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae72da07d97b4d9b9e0d246ff83901ed" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 34185 } } peers { permanent_uuid: "7eecdaa8d1dc4a9988492494fbb17f1d" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 33833 } } peers { permanent_uuid: "a49a0f0a6258477d8126ad3c88bf0d44" member_type: VOTER last_known_addr { host: "127.0.57.196" port: 43761 } } peers { permanent_uuid: "6f2066e97dec4d75a86e72155048de14" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 43041 } attrs { } } I0507 06:31:00.628753 4543 leader_election.cc:296] T 143439bb7e50454aab47fae4ed3ad4ae P 7eecdaa8d1dc4a9988492494fbb17f1d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ae72da07d97b4d9b9e0d246ff83901ed (127.0.57.194:34185), a49a0f0a6258477d8126ad3c88bf0d44 (127.0.57.196:43761), 6f2066e97dec4d75a86e72155048de14 (127.0.57.193:43041) I0507 06:31:00.631280 4440 tablet_service.cc:1286] Received RequestConsensusVote() RPC: tablet_id: "143439bb7e50454aab47fae4ed3ad4ae" candidate_uuid: "7eecdaa8d1dc4a9988492494fbb17f1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "a49a0f0a6258477d8126ad3c88bf0d44" is_pre_election: true I0507 06:31:00.631495 4440 raft_consensus.cc:2346] T 143439bb7e50454aab47fae4ed3ad4ae P a49a0f0a6258477d8126ad3c88bf0d44 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7eecdaa8d1dc4a9988492494fbb17f1d in term 1. W0507 06:31:01.082873 4373 leader_election.cc:341] T 143439bb7e50454aab47fae4ed3ad4ae P a49a0f0a6258477d8126ad3c88bf0d44 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ae72da07d97b4d9b9e0d246ff83901ed (127.0.57.194:34185): Timed out: RequestConsensusVote RPC to 127.0.57.194:34185 timed out after 1.301s (SENT) W0507 06:31:01.083103 4374 leader_election.cc:341] T 143439bb7e50454aab47fae4ed3ad4ae P a49a0f0a6258477d8126ad3c88bf0d44 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6f2066e97dec4d75a86e72155048de14 (127.0.57.193:43041): Timed out: connection negotiation to 127.0.57.193:43041 for RPC RequestConsensusVote timed out after 1.301s (ON_OUTBOUND_QUEUE) I0507 06:31:01.083253 4374 leader_election.cc:310] T 143439bb7e50454aab47fae4ed3ad4ae P a49a0f0a6258477d8126ad3c88bf0d44 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 7eecdaa8d1dc4a9988492494fbb17f1d, a49a0f0a6258477d8126ad3c88bf0d44; no voters: 6f2066e97dec4d75a86e72155048de14, ae72da07d97b4d9b9e0d246ff83901ed I0507 06:31:01.083947 4546 raft_consensus.cc:2588] T 143439bb7e50454aab47fae4ed3ad4ae P a49a0f0a6258477d8126ad3c88bf0d44 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority I0507 06:31:01.233314 4547 raft_consensus.cc:472] T 143439bb7e50454aab47fae4ed3ad4ae P 7eecdaa8d1dc4a9988492494fbb17f1d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ae72da07d97b4d9b9e0d246ff83901ed) I0507 06:31:01.233464 4547 raft_consensus.cc:494] T 143439bb7e50454aab47fae4ed3ad4ae P 7eecdaa8d1dc4a9988492494fbb17f1d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae72da07d97b4d9b9e0d246ff83901ed" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 34185 } } peers { permanent_uuid: "7eecdaa8d1dc4a9988492494fbb17f1d" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 33833 } } peers { permanent_uuid: "a49a0f0a6258477d8126ad3c88bf0d44" member_type: VOTER last_known_addr { host: "127.0.57.196" port: 43761 } } peers { permanent_uuid: "6f2066e97dec4d75a86e72155048de14" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 43041 } attrs { } } W0507 06:31:01.234128 4251 leader_election.cc:341] T 143439bb7e50454aab47fae4ed3ad4ae P 7eecdaa8d1dc4a9988492494fbb17f1d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ae72da07d97b4d9b9e0d246ff83901ed (127.0.57.194:34185): Timed out: connection negotiation to 127.0.57.194:34185 for RPC RequestConsensusVote timed out after 0.604s (ON_OUTBOUND_QUEUE) I0507 06:31:01.234117 4547 leader_election.cc:296] T 143439bb7e50454aab47fae4ed3ad4ae P 7eecdaa8d1dc4a9988492494fbb17f1d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ae72da07d97b4d9b9e0d246ff83901ed (127.0.57.194:34185), a49a0f0a6258477d8126ad3c88bf0d44 (127.0.57.196:43761), 6f2066e97dec4d75a86e72155048de14 (127.0.57.193:43041) W0507 06:31:01.234287 4252 leader_election.cc:341] T 143439bb7e50454aab47fae4ed3ad4ae P 7eecdaa8d1dc4a9988492494fbb17f1d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6f2066e97dec4d75a86e72155048de14 (127.0.57.193:43041): Timed out: connection negotiation to 127.0.57.193:43041 for RPC RequestConsensusVote timed out after 0.604s (ON_OUTBOUND_QUEUE) I0507 06:31:01.234455 4252 leader_election.cc:310] T 143439bb7e50454aab47fae4ed3ad4ae P 7eecdaa8d1dc4a9988492494fbb17f1d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 7eecdaa8d1dc4a9988492494fbb17f1d, a49a0f0a6258477d8126ad3c88bf0d44; no voters: 6f2066e97dec4d75a86e72155048de14, ae72da07d97b4d9b9e0d246ff83901ed I0507 06:31:01.234699 4547 raft_consensus.cc:2588] T 143439bb7e50454aab47fae4ed3ad4ae P 7eecdaa8d1dc4a9988492494fbb17f1d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority I0507 06:31:01.235469 4440 tablet_service.cc:1286] Received RequestConsensusVote() RPC: tablet_id: "143439bb7e50454aab47fae4ed3ad4ae" candidate_uuid: "7eecdaa8d1dc4a9988492494fbb17f1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "a49a0f0a6258477d8126ad3c88bf0d44" is_pre_election: true I0507 06:31:01.235726 4440 raft_consensus.cc:2346] T 143439bb7e50454aab47fae4ed3ad4ae P a49a0f0a6258477d8126ad3c88bf0d44 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7eecdaa8d1dc4a9988492494fbb17f1d in term 1. I0507 06:31:01.463604 4546 raft_consensus.cc:472] T 143439bb7e50454aab47fae4ed3ad4ae P a49a0f0a6258477d8126ad3c88bf0d44 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ae72da07d97b4d9b9e0d246ff83901ed) I0507 06:31:01.463798 4546 raft_consensus.cc:494] T 143439bb7e50454aab47fae4ed3ad4ae P a49a0f0a6258477d8126ad3c88bf0d44 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae72da07d97b4d9b9e0d246ff83901ed" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 34185 } } peers { permanent_uuid: "7eecdaa8d1dc4a9988492494fbb17f1d" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 33833 } } peers { permanent_uuid: "a49a0f0a6258477d8126ad3c88bf0d44" member_type: VOTER last_known_addr { host: "127.0.57.196" port: 43761 } } peers { permanent_uuid: "6f2066e97dec4d75a86e72155048de14" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 43041 } attrs { } } I0507 06:31:01.464501 4546 leader_election.cc:296] T 143439bb7e50454aab47fae4ed3ad4ae P a49a0f0a6258477d8126ad3c88bf0d44 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ae72da07d97b4d9b9e0d246ff83901ed (127.0.57.194:34185), 7eecdaa8d1dc4a9988492494fbb17f1d (127.0.57.195:33833), 6f2066e97dec4d75a86e72155048de14 (127.0.57.193:43041) I0507 06:31:01.465241 4318 tablet_service.cc:1286] Received RequestConsensusVote() RPC: tablet_id: "143439bb7e50454aab47fae4ed3ad4ae" candidate_uuid: "a49a0f0a6258477d8126ad3c88bf0d44" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "7eecdaa8d1dc4a9988492494fbb17f1d" is_pre_election: true I0507 06:31:01.465450 4318 raft_consensus.cc:2346] T 143439bb7e50454aab47fae4ed3ad4ae P 7eecdaa8d1dc4a9988492494fbb17f1d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a49a0f0a6258477d8126ad3c88bf0d44 in term 1. W0507 06:31:01.646751 4373 leader_election.cc:341] T 143439bb7e50454aab47fae4ed3ad4ae P a49a0f0a6258477d8126ad3c88bf0d44 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ae72da07d97b4d9b9e0d246ff83901ed (127.0.57.194:34185): Timed out: RequestConsensusVote RPC to 127.0.57.194:34185 timed out after 1.298s (SENT) W0507 06:31:01.646966 4374 leader_election.cc:341] T 143439bb7e50454aab47fae4ed3ad4ae P a49a0f0a6258477d8126ad3c88bf0d44 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6f2066e97dec4d75a86e72155048de14 (127.0.57.193:43041): Timed out: connection negotiation to 127.0.57.193:43041 for RPC RequestConsensusVote timed out after 1.298s (ON_OUTBOUND_QUEUE) I0507 06:31:01.647094 4374 leader_election.cc:310] T 143439bb7e50454aab47fae4ed3ad4ae P a49a0f0a6258477d8126ad3c88bf0d44 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 7eecdaa8d1dc4a9988492494fbb17f1d, a49a0f0a6258477d8126ad3c88bf0d44; no voters: 6f2066e97dec4d75a86e72155048de14, ae72da07d97b4d9b9e0d246ff83901ed I0507 06:31:01.647538 4546 raft_consensus.cc:2588] T 143439bb7e50454aab47fae4ed3ad4ae P a49a0f0a6258477d8126ad3c88bf0d44 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority W0507 06:31:01.761198 4515 negotiation.cc:313] Failed RPC negotiation. Trace: 0507 06:30:58.568390 (+ 0us) reactor.cc:583] Submitting negotiation task for client connection to 127.0.57.193:43041 0507 06:30:58.568674 (+ 284us) negotiation.cc:98] Waiting for socket to connect 0507 06:30:58.568693 (+ 19us) client_negotiation.cc:167] Beginning negotiation 0507 06:30:58.568838 (+ 145us) client_negotiation.cc:244] Sending NEGOTIATE NegotiatePB request 0507 06:31:01.760929 (+3192091us) negotiation.cc:304] Negotiation complete: Timed out: Client connection negotiation failed: client connection to 127.0.57.193:43041 Metrics: {"client-negotiator.queue_time_us":88} W0507 06:31:01.761555 4374 leader_election.cc:341] T 143439bb7e50454aab47fae4ed3ad4ae P a49a0f0a6258477d8126ad3c88bf0d44 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6f2066e97dec4d75a86e72155048de14 (127.0.57.193:43041): Timed out: Client connection negotiation failed: client connection to 127.0.57.193:43041 W0507 06:31:02.130631 4373 leader_election.cc:341] T 143439bb7e50454aab47fae4ed3ad4ae P a49a0f0a6258477d8126ad3c88bf0d44 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ae72da07d97b4d9b9e0d246ff83901ed (127.0.57.194:34185): Timed out: RequestConsensusVote RPC to 127.0.57.194:34185 timed out after 0.664s (SENT) I0507 06:31:02.130900 4373 leader_election.cc:310] T 143439bb7e50454aab47fae4ed3ad4ae P a49a0f0a6258477d8126ad3c88bf0d44 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 7eecdaa8d1dc4a9988492494fbb17f1d, a49a0f0a6258477d8126ad3c88bf0d44; no voters: 6f2066e97dec4d75a86e72155048de14, ae72da07d97b4d9b9e0d246ff83901ed I0507 06:31:02.131382 4546 raft_consensus.cc:2588] T 143439bb7e50454aab47fae4ed3ad4ae P a49a0f0a6258477d8126ad3c88bf0d44 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority I0507 06:31:02.620949 4546 raft_consensus.cc:472] T 143439bb7e50454aab47fae4ed3ad4ae P a49a0f0a6258477d8126ad3c88bf0d44 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ae72da07d97b4d9b9e0d246ff83901ed) I0507 06:31:02.621178 4546 raft_consensus.cc:494] T 143439bb7e50454aab47fae4ed3ad4ae P a49a0f0a6258477d8126ad3c88bf0d44 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae72da07d97b4d9b9e0d246ff83901ed" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 34185 } } peers { permanent_uuid: "7eecdaa8d1dc4a9988492494fbb17f1d" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 33833 } } peers { permanent_uuid: "a49a0f0a6258477d8126ad3c88bf0d44" member_type: VOTER last_known_addr { host: "127.0.57.196" port: 43761 } } peers { permanent_uuid: "6f2066e97dec4d75a86e72155048de14" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 43041 } attrs { } } I0507 06:31:02.622036 4546 leader_election.cc:296] T 143439bb7e50454aab47fae4ed3ad4ae P a49a0f0a6258477d8126ad3c88bf0d44 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ae72da07d97b4d9b9e0d246ff83901ed (127.0.57.194:34185), 7eecdaa8d1dc4a9988492494fbb17f1d (127.0.57.195:33833), 6f2066e97dec4d75a86e72155048de14 (127.0.57.193:43041) I0507 06:31:02.622905 4318 tablet_service.cc:1286] Received RequestConsensusVote() RPC: tablet_id: "143439bb7e50454aab47fae4ed3ad4ae" candidate_uuid: "a49a0f0a6258477d8126ad3c88bf0d44" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "7eecdaa8d1dc4a9988492494fbb17f1d" is_pre_election: true I0507 06:31:02.623183 4318 raft_consensus.cc:2346] T 143439bb7e50454aab47fae4ed3ad4ae P 7eecdaa8d1dc4a9988492494fbb17f1d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a49a0f0a6258477d8126ad3c88bf0d44 in term 1. W0507 06:31:03.042098 4252 leader_election.cc:341] T 143439bb7e50454aab47fae4ed3ad4ae P 7eecdaa8d1dc4a9988492494fbb17f1d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6f2066e97dec4d75a86e72155048de14 (127.0.57.193:43041): Timed out: connection negotiation to 127.0.57.193:43041 for RPC RequestConsensusVote timed out after 1.807s (ON_OUTBOUND_QUEUE) W0507 06:31:03.042892 4251 leader_election.cc:341] T 143439bb7e50454aab47fae4ed3ad4ae P 7eecdaa8d1dc4a9988492494fbb17f1d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ae72da07d97b4d9b9e0d246ff83901ed (127.0.57.194:34185): Timed out: connection negotiation to 127.0.57.194:34185 for RPC RequestConsensusVote timed out after 1.807s (ON_OUTBOUND_QUEUE) I0507 06:31:03.042966 4251 leader_election.cc:310] T 143439bb7e50454aab47fae4ed3ad4ae P 7eecdaa8d1dc4a9988492494fbb17f1d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 7eecdaa8d1dc4a9988492494fbb17f1d, a49a0f0a6258477d8126ad3c88bf0d44; no voters: 6f2066e97dec4d75a86e72155048de14, ae72da07d97b4d9b9e0d246ff83901ed I0507 06:31:03.043721 4549 raft_consensus.cc:2588] T 143439bb7e50454aab47fae4ed3ad4ae P 7eecdaa8d1dc4a9988492494fbb17f1d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority I0507 06:31:03.690625 4530 raft_consensus.cc:472] T 143439bb7e50454aab47fae4ed3ad4ae P 6f2066e97dec4d75a86e72155048de14 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ae72da07d97b4d9b9e0d246ff83901ed) I0507 06:31:03.691776 4530 raft_consensus.cc:494] T 143439bb7e50454aab47fae4ed3ad4ae P 6f2066e97dec4d75a86e72155048de14 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae72da07d97b4d9b9e0d246ff83901ed" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 34185 } } peers { permanent_uuid: "7eecdaa8d1dc4a9988492494fbb17f1d" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 33833 } } peers { permanent_uuid: "a49a0f0a6258477d8126ad3c88bf0d44" member_type: VOTER last_known_addr { host: "127.0.57.196" port: 43761 } } peers { permanent_uuid: "6f2066e97dec4d75a86e72155048de14" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 43041 } attrs { } } I0507 06:31:03.707191 4530 leader_election.cc:296] T 143439bb7e50454aab47fae4ed3ad4ae P 6f2066e97dec4d75a86e72155048de14 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ae72da07d97b4d9b9e0d246ff83901ed (127.0.57.194:34185), 7eecdaa8d1dc4a9988492494fbb17f1d (127.0.57.195:33833), a49a0f0a6258477d8126ad3c88bf0d44 (127.0.57.196:43761) W0507 06:31:03.713526 4545 negotiation.cc:313] Failed RPC negotiation. Trace: 0507 06:31:00.629080 (+ 0us) reactor.cc:583] Submitting negotiation task for client connection to 127.0.57.193:43041 0507 06:31:00.630458 (+ 1378us) negotiation.cc:98] Waiting for socket to connect 0507 06:31:00.630469 (+ 11us) client_negotiation.cc:167] Beginning negotiation 0507 06:31:00.630523 (+ 54us) client_negotiation.cc:244] Sending NEGOTIATE NegotiatePB request 0507 06:31:03.713269 (+3082746us) negotiation.cc:304] Negotiation complete: Timed out: Client connection negotiation failed: client connection to 127.0.57.193:43041 Metrics: {"client-negotiator.queue_time_us":1301,"thread_start_us":1457,"threads_started":1} W0507 06:31:03.715673 4519 negotiation.cc:313] Failed RPC negotiation. Trace: 0507 06:31:03.692329 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.0.57.195:57939 0507 06:31:03.706755 (+ 14426us) server_negotiation.cc:184] Beginning negotiation 0507 06:31:03.706759 (+ 4us) server_negotiation.cc:373] Waiting for connection header 0507 06:31:03.706778 (+ 19us) server_negotiation.cc:381] Connection header received 0507 06:31:03.706825 (+ 47us) server_negotiation.cc:337] Received NEGOTIATE NegotiatePB request 0507 06:31:03.706830 (+ 5us) server_negotiation.cc:420] Received NEGOTIATE request from client 0507 06:31:03.706896 (+ 66us) server_negotiation.cc:349] Sending NEGOTIATE NegotiatePB response 0507 06:31:03.707060 (+ 164us) negotiation.cc:304] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.57.195:57939: BlockingWrite error: write error: Broken pipe (error 32) Metrics: {"server-negotiator.queue_time_us":14038,"threads_started":1} W0507 06:31:03.720266 4550 negotiation.cc:313] Failed RPC negotiation. Trace: 0507 06:31:03.688507 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.0.57.196:42943 0507 06:31:03.705298 (+ 16791us) server_negotiation.cc:184] Beginning negotiation 0507 06:31:03.705301 (+ 3us) server_negotiation.cc:373] Waiting for connection header 0507 06:31:03.705442 (+ 141us) server_negotiation.cc:381] Connection header received 0507 06:31:03.705505 (+ 63us) server_negotiation.cc:337] Received NEGOTIATE NegotiatePB request 0507 06:31:03.705510 (+ 5us) server_negotiation.cc:420] Received NEGOTIATE request from client 0507 06:31:03.705603 (+ 93us) server_negotiation.cc:349] Sending NEGOTIATE NegotiatePB response 0507 06:31:03.711924 (+ 6321us) negotiation.cc:304] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.57.196:42943: BlockingWrite error: write error: Broken pipe (error 32) Metrics: {"server-negotiator.queue_time_us":16150,"thread_start_us":3769,"threads_started":1} W0507 06:31:03.722096 4552 negotiation.cc:313] Failed RPC negotiation. Trace: 0507 06:31:03.721367 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.0.57.196:60979 0507 06:31:03.721558 (+ 191us) server_negotiation.cc:184] Beginning negotiation 0507 06:31:03.721562 (+ 4us) server_negotiation.cc:373] Waiting for connection header 0507 06:31:03.721577 (+ 15us) server_negotiation.cc:381] Connection header received 0507 06:31:03.721613 (+ 36us) server_negotiation.cc:337] Received NEGOTIATE NegotiatePB request 0507 06:31:03.721616 (+ 3us) server_negotiation.cc:420] Received NEGOTIATE request from client 0507 06:31:03.721665 (+ 49us) server_negotiation.cc:349] Sending NEGOTIATE NegotiatePB response 0507 06:31:03.721896 (+ 231us) negotiation.cc:304] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.57.196:60979: BlockingWrite error: write error: Broken pipe (error 32) Metrics: {"server-negotiator.queue_time_us":68} I0507 06:31:03.722146 4318 tablet_service.cc:1286] Received RequestConsensusVote() RPC: tablet_id: "143439bb7e50454aab47fae4ed3ad4ae" candidate_uuid: "6f2066e97dec4d75a86e72155048de14" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "7eecdaa8d1dc4a9988492494fbb17f1d" is_pre_election: true I0507 06:31:03.722451 4318 raft_consensus.cc:2346] T 143439bb7e50454aab47fae4ed3ad4ae P 7eecdaa8d1dc4a9988492494fbb17f1d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6f2066e97dec4d75a86e72155048de14 in term 1. I0507 06:31:03.722518 4072 tablet_service.cc:1286] Received RequestConsensusVote() RPC: tablet_id: "143439bb7e50454aab47fae4ed3ad4ae" candidate_uuid: "a49a0f0a6258477d8126ad3c88bf0d44" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "6f2066e97dec4d75a86e72155048de14" is_pre_election: true I0507 06:31:03.722815 4072 raft_consensus.cc:2346] T 143439bb7e50454aab47fae4ed3ad4ae P 6f2066e97dec4d75a86e72155048de14 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a49a0f0a6258477d8126ad3c88bf0d44 in term 1. W0507 06:31:03.724334 4551 negotiation.cc:313] Failed RPC negotiation. Trace: 0507 06:31:03.690127 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.0.57.195:34313 0507 06:31:03.706292 (+ 16165us) server_negotiation.cc:184] Beginning negotiation 0507 06:31:03.706296 (+ 4us) server_negotiation.cc:373] Waiting for connection header 0507 06:31:03.712954 (+ 6658us) server_negotiation.cc:381] Connection header received 0507 06:31:03.713014 (+ 60us) server_negotiation.cc:337] Received NEGOTIATE NegotiatePB request 0507 06:31:03.713019 (+ 5us) server_negotiation.cc:420] Received NEGOTIATE request from client 0507 06:31:03.713079 (+ 60us) server_negotiation.cc:349] Sending NEGOTIATE NegotiatePB response 0507 06:31:03.713164 (+ 85us) server_negotiation.cc:205] Negotiated authn=CERTIFICATE 0507 06:31:03.724135 (+ 10971us) negotiation.cc:304] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.57.195:34313: BlockingRecv error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107) Metrics: {"server-negotiator.queue_time_us":14735,"thread_start_us":13402,"threads_started":1} I0507 06:31:03.724773 4374 leader_election.cc:310] T 143439bb7e50454aab47fae4ed3ad4ae P a49a0f0a6258477d8126ad3c88bf0d44 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 6f2066e97dec4d75a86e72155048de14, 7eecdaa8d1dc4a9988492494fbb17f1d, a49a0f0a6258477d8126ad3c88bf0d44; no voters: I0507 06:31:03.725440 4557 raft_consensus.cc:2640] T 143439bb7e50454aab47fae4ed3ad4ae P a49a0f0a6258477d8126ad3c88bf0d44 [term 1 FOLLOWER]: Leader pre-election won for term 2 I0507 06:31:03.725524 4557 raft_consensus.cc:472] T 143439bb7e50454aab47fae4ed3ad4ae P a49a0f0a6258477d8126ad3c88bf0d44 [term 1 FOLLOWER]: Starting leader election (detected failure of leader ae72da07d97b4d9b9e0d246ff83901ed) I0507 06:31:03.725687 4557 raft_consensus.cc:2886] T 143439bb7e50454aab47fae4ed3ad4ae P a49a0f0a6258477d8126ad3c88bf0d44 [term 1 FOLLOWER]: Advancing to term 2 I0507 06:31:03.726351 4440 tablet_service.cc:1286] Received RequestConsensusVote() RPC: tablet_id: "143439bb7e50454aab47fae4ed3ad4ae" candidate_uuid: "6f2066e97dec4d75a86e72155048de14" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "a49a0f0a6258477d8126ad3c88bf0d44" is_pre_election: true I0507 06:31:03.727840 4557 raft_consensus.cc:494] T 143439bb7e50454aab47fae4ed3ad4ae P a49a0f0a6258477d8126ad3c88bf0d44 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae72da07d97b4d9b9e0d246ff83901ed" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 34185 } } peers { permanent_uuid: "7eecdaa8d1dc4a9988492494fbb17f1d" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 33833 } } peers { permanent_uuid: "a49a0f0a6258477d8126ad3c88bf0d44" member_type: VOTER last_known_addr { host: "127.0.57.196" port: 43761 } } peers { permanent_uuid: "6f2066e97dec4d75a86e72155048de14" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 43041 } attrs { } } I0507 06:31:03.728361 4440 raft_consensus.cc:2276] T 143439bb7e50454aab47fae4ed3ad4ae P a49a0f0a6258477d8126ad3c88bf0d44 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6f2066e97dec4d75a86e72155048de14 in current term 2: Already voted for candidate a49a0f0a6258477d8126ad3c88bf0d44 in this term. I0507 06:31:03.728863 4557 leader_election.cc:296] T 143439bb7e50454aab47fae4ed3ad4ae P a49a0f0a6258477d8126ad3c88bf0d44 [CANDIDATE]: Term 2 election: Requested vote from peers ae72da07d97b4d9b9e0d246ff83901ed (127.0.57.194:34185), 7eecdaa8d1dc4a9988492494fbb17f1d (127.0.57.195:33833), 6f2066e97dec4d75a86e72155048de14 (127.0.57.193:43041) I0507 06:31:03.729321 4318 tablet_service.cc:1286] Received RequestConsensusVote() RPC: tablet_id: "143439bb7e50454aab47fae4ed3ad4ae" candidate_uuid: "a49a0f0a6258477d8126ad3c88bf0d44" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "7eecdaa8d1dc4a9988492494fbb17f1d" I0507 06:31:03.729375 4072 tablet_service.cc:1286] Received RequestConsensusVote() RPC: tablet_id: "143439bb7e50454aab47fae4ed3ad4ae" candidate_uuid: "a49a0f0a6258477d8126ad3c88bf0d44" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "6f2066e97dec4d75a86e72155048de14" I0507 06:31:03.729511 4318 raft_consensus.cc:2886] T 143439bb7e50454aab47fae4ed3ad4ae P 7eecdaa8d1dc4a9988492494fbb17f1d [term 1 FOLLOWER]: Advancing to term 2 I0507 06:31:03.729522 4072 raft_consensus.cc:2886] T 143439bb7e50454aab47fae4ed3ad4ae P 6f2066e97dec4d75a86e72155048de14 [term 1 FOLLOWER]: Advancing to term 2 I0507 06:31:03.731195 4318 raft_consensus.cc:2346] T 143439bb7e50454aab47fae4ed3ad4ae P 7eecdaa8d1dc4a9988492494fbb17f1d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a49a0f0a6258477d8126ad3c88bf0d44 in term 2. I0507 06:31:03.731314 4072 raft_consensus.cc:2346] T 143439bb7e50454aab47fae4ed3ad4ae P 6f2066e97dec4d75a86e72155048de14 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a49a0f0a6258477d8126ad3c88bf0d44 in term 2. I0507 06:31:03.731622 4374 leader_election.cc:310] T 143439bb7e50454aab47fae4ed3ad4ae P a49a0f0a6258477d8126ad3c88bf0d44 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 6f2066e97dec4d75a86e72155048de14, 7eecdaa8d1dc4a9988492494fbb17f1d, a49a0f0a6258477d8126ad3c88bf0d44; no voters: I0507 06:31:03.731864 4557 raft_consensus.cc:2640] T 143439bb7e50454aab47fae4ed3ad4ae P a49a0f0a6258477d8126ad3c88bf0d44 [term 2 FOLLOWER]: Leader election won for term 2 I0507 06:31:03.732265 4557 raft_consensus.cc:667] T 143439bb7e50454aab47fae4ed3ad4ae P a49a0f0a6258477d8126ad3c88bf0d44 [term 2 LEADER]: Becoming Leader. State: Replica: a49a0f0a6258477d8126ad3c88bf0d44, State: Running, Role: LEADER I0507 06:31:03.732723 4557 consensus_queue.cc:206] T 143439bb7e50454aab47fae4ed3ad4ae P a49a0f0a6258477d8126ad3c88bf0d44 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae72da07d97b4d9b9e0d246ff83901ed" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 34185 } } peers { permanent_uuid: "7eecdaa8d1dc4a9988492494fbb17f1d" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 33833 } } peers { permanent_uuid: "a49a0f0a6258477d8126ad3c88bf0d44" member_type: VOTER last_known_addr { host: "127.0.57.196" port: 43761 } } peers { permanent_uuid: "6f2066e97dec4d75a86e72155048de14" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 43041 } attrs { } } I0507 06:31:03.735797 3953 catalog_manager.cc:3988] T 143439bb7e50454aab47fae4ed3ad4ae P a49a0f0a6258477d8126ad3c88bf0d44 reported cstate change: term changed from 1 to 2, leader changed from ae72da07d97b4d9b9e0d246ff83901ed (127.0.57.194) to a49a0f0a6258477d8126ad3c88bf0d44 (127.0.57.196). New cstate: current_term: 2 leader_uuid: "a49a0f0a6258477d8126ad3c88bf0d44" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae72da07d97b4d9b9e0d246ff83901ed" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 34185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7eecdaa8d1dc4a9988492494fbb17f1d" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 33833 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a49a0f0a6258477d8126ad3c88bf0d44" member_type: VOTER last_known_addr { host: "127.0.57.196" port: 43761 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6f2066e97dec4d75a86e72155048de14" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 43041 } attrs { } health_report { overall_health: UNKNOWN } } } I0507 06:31:03.779549 4318 raft_consensus.cc:1181] T 143439bb7e50454aab47fae4ed3ad4ae P 7eecdaa8d1dc4a9988492494fbb17f1d [term 2 FOLLOWER]: Refusing update from remote peer a49a0f0a6258477d8126ad3c88bf0d44: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch) I0507 06:31:03.780357 4557 consensus_queue.cc:984] T 143439bb7e50454aab47fae4ed3ad4ae P a49a0f0a6258477d8126ad3c88bf0d44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7eecdaa8d1dc4a9988492494fbb17f1d" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 33833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s I0507 06:31:03.786576 4072 raft_consensus.cc:1181] T 143439bb7e50454aab47fae4ed3ad4ae P 6f2066e97dec4d75a86e72155048de14 [term 2 FOLLOWER]: Refusing update from remote peer a49a0f0a6258477d8126ad3c88bf0d44: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch) I0507 06:31:03.787278 4557 consensus_queue.cc:984] T 143439bb7e50454aab47fae4ed3ad4ae P a49a0f0a6258477d8126ad3c88bf0d44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f2066e97dec4d75a86e72155048de14" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 43041 } attrs { }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s I0507 06:31:03.795862 4184 tablet_service.cc:1286] Received RequestConsensusVote() RPC: tablet_id: "143439bb7e50454aab47fae4ed3ad4ae" candidate_uuid: "6f2066e97dec4d75a86e72155048de14" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "ae72da07d97b4d9b9e0d246ff83901ed" is_pre_election: true I0507 06:31:03.796406 4184 raft_consensus.cc:2307] T 143439bb7e50454aab47fae4ed3ad4ae P ae72da07d97b4d9b9e0d246ff83901ed [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate 6f2066e97dec4d75a86e72155048de14 for term 2 because replica is either leader or believes a valid leader to be alive. I0507 06:31:03.797184 4006 leader_election.cc:310] T 143439bb7e50454aab47fae4ed3ad4ae P 6f2066e97dec4d75a86e72155048de14 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 6f2066e97dec4d75a86e72155048de14, 7eecdaa8d1dc4a9988492494fbb17f1d; no voters: a49a0f0a6258477d8126ad3c88bf0d44, ae72da07d97b4d9b9e0d246ff83901ed I0507 06:31:03.798075 4530 raft_consensus.cc:2588] T 143439bb7e50454aab47fae4ed3ad4ae P 6f2066e97dec4d75a86e72155048de14 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority I0507 06:31:03.799962 4318 raft_consensus.cc:1149] T 143439bb7e50454aab47fae4ed3ad4ae P 7eecdaa8d1dc4a9988492494fbb17f1d [term 2 FOLLOWER]: Rejecting Update request from peer ae72da07d97b4d9b9e0d246ff83901ed for earlier term 1. Current term is 2. Ops: [] I0507 06:31:03.800319 4196 tablet_service.cc:1286] Received RequestConsensusVote() RPC: tablet_id: "143439bb7e50454aab47fae4ed3ad4ae" candidate_uuid: "a49a0f0a6258477d8126ad3c88bf0d44" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "ae72da07d97b4d9b9e0d246ff83901ed" is_pre_election: true I0507 06:31:03.804090 4196 raft_consensus.cc:2321] T 143439bb7e50454aab47fae4ed3ad4ae P ae72da07d97b4d9b9e0d246ff83901ed [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate a49a0f0a6258477d8126ad3c88bf0d44 for term 2 because replica is already servicing an update from a current leader or another vote. I0507 06:31:03.808482 4072 raft_consensus.cc:1149] T 143439bb7e50454aab47fae4ed3ad4ae P 6f2066e97dec4d75a86e72155048de14 [term 2 FOLLOWER]: Rejecting Update request from peer ae72da07d97b4d9b9e0d246ff83901ed for earlier term 1. Current term is 2. Ops: [] W0507 06:31:03.813118 4544 negotiation.cc:313] Failed RPC negotiation. Trace: 0507 06:31:00.629028 (+ 0us) reactor.cc:583] Submitting negotiation task for client connection to 127.0.57.194:34185 0507 06:31:00.630005 (+ 977us) negotiation.cc:98] Waiting for socket to connect 0507 06:31:00.630025 (+ 20us) client_negotiation.cc:167] Beginning negotiation 0507 06:31:00.630155 (+ 130us) client_negotiation.cc:244] Sending NEGOTIATE NegotiatePB request 0507 06:31:03.812817 (+3182662us) negotiation.cc:304] Negotiation complete: Timed out: Client connection negotiation failed: client connection to 127.0.57.194:34185 Metrics: {"client-negotiator.queue_time_us":839,"thread_start_us":710,"threads_started":1} I0507 06:31:03.807571 4186 tablet_service.cc:1286] Received RequestConsensusVote() RPC: tablet_id: "143439bb7e50454aab47fae4ed3ad4ae" candidate_uuid: "a49a0f0a6258477d8126ad3c88bf0d44" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "ae72da07d97b4d9b9e0d246ff83901ed" I0507 06:31:03.803159 4189 tablet_service.cc:1286] Received RequestConsensusVote() RPC: tablet_id: "143439bb7e50454aab47fae4ed3ad4ae" candidate_uuid: "a49a0f0a6258477d8126ad3c88bf0d44" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "ae72da07d97b4d9b9e0d246ff83901ed" is_pre_election: true I0507 06:31:03.804323 4187 tablet_service.cc:1286] Received RequestConsensusVote() RPC: tablet_id: "143439bb7e50454aab47fae4ed3ad4ae" candidate_uuid: "a49a0f0a6258477d8126ad3c88bf0d44" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "ae72da07d97b4d9b9e0d246ff83901ed" is_pre_election: true I0507 06:31:03.808055 4193 tablet_service.cc:1286] Received RequestConsensusVote() RPC: tablet_id: "143439bb7e50454aab47fae4ed3ad4ae" candidate_uuid: "a49a0f0a6258477d8126ad3c88bf0d44" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "ae72da07d97b4d9b9e0d246ff83901ed" is_pre_election: true I0507 06:31:03.805177 4190 tablet_service.cc:1286] Received RequestConsensusVote() RPC: tablet_id: "143439bb7e50454aab47fae4ed3ad4ae" candidate_uuid: "a49a0f0a6258477d8126ad3c88bf0d44" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "ae72da07d97b4d9b9e0d246ff83901ed" is_pre_election: true I0507 06:31:03.816105 4532 consensus_queue.cc:995] T 143439bb7e50454aab47fae4ed3ad4ae P ae72da07d97b4d9b9e0d246ff83901ed [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7eecdaa8d1dc4a9988492494fbb17f1d" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 33833 }, Status: INVALID_TERM, Last received: 1.3, Next index: 4, Last known committed idx: 4, Time since last communication: 0.014s I0507 06:31:03.805624 4192 tablet_service.cc:1286] Received RequestConsensusVote() RPC: tablet_id: "143439bb7e50454aab47fae4ed3ad4ae" candidate_uuid: "a49a0f0a6258477d8126ad3c88bf0d44" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "ae72da07d97b4d9b9e0d246ff83901ed" is_pre_election: true I0507 06:31:03.814348 4188 tablet_service.cc:1286] Received RequestConsensusVote() RPC: tablet_id: "143439bb7e50454aab47fae4ed3ad4ae" candidate_uuid: "a49a0f0a6258477d8126ad3c88bf0d44" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "ae72da07d97b4d9b9e0d246ff83901ed" is_pre_election: true I0507 06:31:03.819281 4524 consensus_queue.cc:995] T 143439bb7e50454aab47fae4ed3ad4ae P ae72da07d97b4d9b9e0d246ff83901ed [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6f2066e97dec4d75a86e72155048de14" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 43041 } attrs { }, Status: INVALID_TERM, Last received: 1.3, Next index: 4, Last known committed idx: 4, Time since last communication: 0.001s I0507 06:31:03.815457 4195 tablet_service.cc:1286] Received RequestConsensusVote() RPC: tablet_id: "143439bb7e50454aab47fae4ed3ad4ae" candidate_uuid: "a49a0f0a6258477d8126ad3c88bf0d44" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "ae72da07d97b4d9b9e0d246ff83901ed" is_pre_election: true I0507 06:31:03.819831 4185 raft_consensus.cc:2881] T 143439bb7e50454aab47fae4ed3ad4ae P ae72da07d97b4d9b9e0d246ff83901ed [term 1 LEADER]: Stepping down as leader of term 1 I0507 06:31:03.819924 4185 raft_consensus.cc:705] T 143439bb7e50454aab47fae4ed3ad4ae P ae72da07d97b4d9b9e0d246ff83901ed [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ae72da07d97b4d9b9e0d246ff83901ed, State: Running, Role: LEADER I0507 06:31:03.815662 4191 tablet_service.cc:1286] Received RequestConsensusVote() RPC: tablet_id: "143439bb7e50454aab47fae4ed3ad4ae" candidate_uuid: "a49a0f0a6258477d8126ad3c88bf0d44" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "ae72da07d97b4d9b9e0d246ff83901ed" is_pre_election: true I0507 06:31:03.816321 4194 tablet_service.cc:1286] Received RequestConsensusVote() RPC: tablet_id: "143439bb7e50454aab47fae4ed3ad4ae" candidate_uuid: "a49a0f0a6258477d8126ad3c88bf0d44" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "ae72da07d97b4d9b9e0d246ff83901ed" is_pre_election: true I0507 06:31:03.820231 4185 consensus_queue.cc:229] T 143439bb7e50454aab47fae4ed3ad4ae P ae72da07d97b4d9b9e0d246ff83901ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae72da07d97b4d9b9e0d246ff83901ed" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 34185 } } peers { permanent_uuid: "7eecdaa8d1dc4a9988492494fbb17f1d" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 33833 } } peers { permanent_uuid: "a49a0f0a6258477d8126ad3c88bf0d44" member_type: VOTER last_known_addr { host: "127.0.57.196" port: 43761 } } peers { permanent_uuid: "6f2066e97dec4d75a86e72155048de14" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 43041 } attrs { } } W0507 06:31:03.822525 4571 negotiation.cc:313] Failed RPC negotiation. Trace: 0507 06:31:03.804721 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.0.57.195:58187 0507 06:31:03.819951 (+ 15230us) server_negotiation.cc:184] Beginning negotiation 0507 06:31:03.819956 (+ 5us) server_negotiation.cc:373] Waiting for connection header 0507 06:31:03.819991 (+ 35us) server_negotiation.cc:381] Connection header received 0507 06:31:03.820038 (+ 47us) server_negotiation.cc:337] Received NEGOTIATE NegotiatePB request 0507 06:31:03.820042 (+ 4us) server_negotiation.cc:420] Received NEGOTIATE request from client 0507 06:31:03.820102 (+ 60us) server_negotiation.cc:349] Sending NEGOTIATE NegotiatePB response 0507 06:31:03.820550 (+ 448us) negotiation.cc:304] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.57.195:58187: BlockingWrite error: write error: Broken pipe (error 32) Metrics: {"server-negotiator.queue_time_us":14136,"thread_start_us":11771,"threads_started":1} I0507 06:31:03.822535 4185 raft_consensus.cc:2886] T 143439bb7e50454aab47fae4ed3ad4ae P ae72da07d97b4d9b9e0d246ff83901ed [term 1 FOLLOWER]: Advancing to term 2 W0507 06:31:03.824884 4570 negotiation.cc:313] Failed RPC negotiation. Trace: 0507 06:31:03.798932 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.0.57.195:48517 0507 06:31:03.815363 (+ 16431us) server_negotiation.cc:184] Beginning negotiation 0507 06:31:03.815370 (+ 7us) server_negotiation.cc:373] Waiting for connection header 0507 06:31:03.815626 (+ 256us) server_negotiation.cc:381] Connection header received 0507 06:31:03.815692 (+ 66us) server_negotiation.cc:337] Received NEGOTIATE NegotiatePB request 0507 06:31:03.815696 (+ 4us) server_negotiation.cc:420] Received NEGOTIATE request from client 0507 06:31:03.815789 (+ 93us) server_negotiation.cc:349] Sending NEGOTIATE NegotiatePB response 0507 06:31:03.817739 (+ 1950us) negotiation.cc:304] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.57.195:48517: BlockingWrite error: write error: Broken pipe (error 32) Metrics: {"server-negotiator.queue_time_us":16006,"thread_start_us":7884,"threads_started":1} I0507 06:31:03.824919 4185 raft_consensus.cc:1181] T 143439bb7e50454aab47fae4ed3ad4ae P ae72da07d97b4d9b9e0d246ff83901ed [term 2 FOLLOWER]: Refusing update from remote peer a49a0f0a6258477d8126ad3c88bf0d44: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch) I0507 06:31:03.825141 4186 raft_consensus.cc:2321] T 143439bb7e50454aab47fae4ed3ad4ae P ae72da07d97b4d9b9e0d246ff83901ed [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate a49a0f0a6258477d8126ad3c88bf0d44 for term 2 because replica is already servicing an update from a current leader or another vote. I0507 06:31:03.825383 4189 raft_consensus.cc:2321] T 143439bb7e50454aab47fae4ed3ad4ae P ae72da07d97b4d9b9e0d246ff83901ed [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a49a0f0a6258477d8126ad3c88bf0d44 for term 2 because replica is already servicing an update from a current leader or another vote. I0507 06:31:03.825619 4187 raft_consensus.cc:2321] T 143439bb7e50454aab47fae4ed3ad4ae P ae72da07d97b4d9b9e0d246ff83901ed [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a49a0f0a6258477d8126ad3c88bf0d44 for term 2 because replica is already servicing an update from a current leader or another vote. I0507 06:31:03.825733 4192 raft_consensus.cc:2321] T 143439bb7e50454aab47fae4ed3ad4ae P ae72da07d97b4d9b9e0d246ff83901ed [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a49a0f0a6258477d8126ad3c88bf0d44 for term 2 because replica is already servicing an update from a current leader or another vote. I0507 06:31:03.825865 4193 raft_consensus.cc:2321] T 143439bb7e50454aab47fae4ed3ad4ae P ae72da07d97b4d9b9e0d246ff83901ed [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a49a0f0a6258477d8126ad3c88bf0d44 for term 2 because replica is already servicing an update from a current leader or another vote. I0507 06:31:03.826113 4195 raft_consensus.cc:2321] T 143439bb7e50454aab47fae4ed3ad4ae P ae72da07d97b4d9b9e0d246ff83901ed [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a49a0f0a6258477d8126ad3c88bf0d44 for term 2 because replica is already servicing an update from a current leader or another vote. W0507 06:31:03.826514 4517 raft_consensus.cc:847] Couldn't advance consensus term.: Illegal state: Can't advance term to: 2 current term: 2 is higher. I0507 06:31:03.826907 4194 raft_consensus.cc:2321] T 143439bb7e50454aab47fae4ed3ad4ae P ae72da07d97b4d9b9e0d246ff83901ed [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a49a0f0a6258477d8126ad3c88bf0d44 for term 2 because replica is already servicing an update from a current leader or another vote. I0507 06:31:03.827118 4191 raft_consensus.cc:2321] T 143439bb7e50454aab47fae4ed3ad4ae P ae72da07d97b4d9b9e0d246ff83901ed [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a49a0f0a6258477d8126ad3c88bf0d44 for term 2 because replica is already servicing an update from a current leader or another vote. I0507 06:31:03.827246 4188 raft_consensus.cc:2321] T 143439bb7e50454aab47fae4ed3ad4ae P ae72da07d97b4d9b9e0d246ff83901ed [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a49a0f0a6258477d8126ad3c88bf0d44 for term 2 because replica is already servicing an update from a current leader or another vote. I0507 06:31:03.827638 4190 raft_consensus.cc:2321] T 143439bb7e50454aab47fae4ed3ad4ae P ae72da07d97b4d9b9e0d246ff83901ed [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a49a0f0a6258477d8126ad3c88bf0d44 for term 2 because replica is already servicing an update from a current leader or another vote. I0507 06:31:03.943840 4564 raft_consensus.cc:472] T 143439bb7e50454aab47fae4ed3ad4ae P 7eecdaa8d1dc4a9988492494fbb17f1d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader a49a0f0a6258477d8126ad3c88bf0d44) I0507 06:31:03.944125 4564 raft_consensus.cc:494] T 143439bb7e50454aab47fae4ed3ad4ae P 7eecdaa8d1dc4a9988492494fbb17f1d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae72da07d97b4d9b9e0d246ff83901ed" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 34185 } } peers { permanent_uuid: "7eecdaa8d1dc4a9988492494fbb17f1d" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 33833 } } peers { permanent_uuid: "a49a0f0a6258477d8126ad3c88bf0d44" member_type: VOTER last_known_addr { host: "127.0.57.196" port: 43761 } } peers { permanent_uuid: "6f2066e97dec4d75a86e72155048de14" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 43041 } attrs { } } I0507 06:31:03.945161 4564 leader_election.cc:296] T 143439bb7e50454aab47fae4ed3ad4ae P 7eecdaa8d1dc4a9988492494fbb17f1d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ae72da07d97b4d9b9e0d246ff83901ed (127.0.57.194:34185), a49a0f0a6258477d8126ad3c88bf0d44 (127.0.57.196:43761), 6f2066e97dec4d75a86e72155048de14 (127.0.57.193:43041) I0507 06:31:03.952081 4072 tablet_service.cc:1286] Received RequestConsensusVote() RPC: tablet_id: "143439bb7e50454aab47fae4ed3ad4ae" candidate_uuid: "7eecdaa8d1dc4a9988492494fbb17f1d" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "6f2066e97dec4d75a86e72155048de14" is_pre_election: true I0507 06:31:03.952101 4190 tablet_service.cc:1286] Received RequestConsensusVote() RPC: tablet_id: "143439bb7e50454aab47fae4ed3ad4ae" candidate_uuid: "7eecdaa8d1dc4a9988492494fbb17f1d" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "ae72da07d97b4d9b9e0d246ff83901ed" is_pre_election: true I0507 06:31:03.952364 4072 raft_consensus.cc:2346] T 143439bb7e50454aab47fae4ed3ad4ae P 6f2066e97dec4d75a86e72155048de14 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7eecdaa8d1dc4a9988492494fbb17f1d in term 2. I0507 06:31:03.952371 4190 raft_consensus.cc:2346] T 143439bb7e50454aab47fae4ed3ad4ae P ae72da07d97b4d9b9e0d246ff83901ed [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7eecdaa8d1dc4a9988492494fbb17f1d in term 2. I0507 06:31:03.952796 4251 leader_election.cc:310] T 143439bb7e50454aab47fae4ed3ad4ae P 7eecdaa8d1dc4a9988492494fbb17f1d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 6f2066e97dec4d75a86e72155048de14, 7eecdaa8d1dc4a9988492494fbb17f1d, ae72da07d97b4d9b9e0d246ff83901ed; no voters: I0507 06:31:03.953109 4564 raft_consensus.cc:2640] T 143439bb7e50454aab47fae4ed3ad4ae P 7eecdaa8d1dc4a9988492494fbb17f1d [term 2 FOLLOWER]: Leader pre-election won for term 3 I0507 06:31:03.953194 4564 raft_consensus.cc:472] T 143439bb7e50454aab47fae4ed3ad4ae P 7eecdaa8d1dc4a9988492494fbb17f1d [term 2 FOLLOWER]: Starting leader election (detected failure of leader a49a0f0a6258477d8126ad3c88bf0d44) I0507 06:31:03.953286 4564 raft_consensus.cc:2886] T 143439bb7e50454aab47fae4ed3ad4ae P 7eecdaa8d1dc4a9988492494fbb17f1d [term 2 FOLLOWER]: Advancing to term 3 I0507 06:31:03.955210 4564 raft_consensus.cc:494] T 143439bb7e50454aab47fae4ed3ad4ae P 7eecdaa8d1dc4a9988492494fbb17f1d [term 3 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae72da07d97b4d9b9e0d246ff83901ed" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 34185 } } peers { permanent_uuid: "7eecdaa8d1dc4a9988492494fbb17f1d" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 33833 } } peers { permanent_uuid: "a49a0f0a6258477d8126ad3c88bf0d44" member_type: VOTER last_known_addr { host: "127.0.57.196" port: 43761 } } peers { permanent_uuid: "6f2066e97dec4d75a86e72155048de14" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 43041 } attrs { } } I0507 06:31:03.955807 4564 leader_election.cc:296] T 143439bb7e50454aab47fae4ed3ad4ae P 7eecdaa8d1dc4a9988492494fbb17f1d [CANDIDATE]: Term 3 election: Requested vote from peers ae72da07d97b4d9b9e0d246ff83901ed (127.0.57.194:34185), a49a0f0a6258477d8126ad3c88bf0d44 (127.0.57.196:43761), 6f2066e97dec4d75a86e72155048de14 (127.0.57.193:43041) I0507 06:31:03.956292 4072 tablet_service.cc:1286] Received RequestConsensusVote() RPC: tablet_id: "143439bb7e50454aab47fae4ed3ad4ae" candidate_uuid: "7eecdaa8d1dc4a9988492494fbb17f1d" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "6f2066e97dec4d75a86e72155048de14" I0507 06:31:03.956312 4190 tablet_service.cc:1286] Received RequestConsensusVote() RPC: tablet_id: "143439bb7e50454aab47fae4ed3ad4ae" candidate_uuid: "7eecdaa8d1dc4a9988492494fbb17f1d" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "ae72da07d97b4d9b9e0d246ff83901ed" I0507 06:31:03.956478 4072 raft_consensus.cc:2886] T 143439bb7e50454aab47fae4ed3ad4ae P 6f2066e97dec4d75a86e72155048de14 [term 2 FOLLOWER]: Advancing to term 3 I0507 06:31:03.956482 4190 raft_consensus.cc:2886] T 143439bb7e50454aab47fae4ed3ad4ae P ae72da07d97b4d9b9e0d246ff83901ed [term 2 FOLLOWER]: Advancing to term 3 I0507 06:31:03.958987 4072 raft_consensus.cc:2346] T 143439bb7e50454aab47fae4ed3ad4ae P 6f2066e97dec4d75a86e72155048de14 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7eecdaa8d1dc4a9988492494fbb17f1d in term 3. I0507 06:31:03.958997 4190 raft_consensus.cc:2346] T 143439bb7e50454aab47fae4ed3ad4ae P ae72da07d97b4d9b9e0d246ff83901ed [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7eecdaa8d1dc4a9988492494fbb17f1d in term 3. I0507 06:31:03.959419 4252 leader_election.cc:310] T 143439bb7e50454aab47fae4ed3ad4ae P 7eecdaa8d1dc4a9988492494fbb17f1d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 6f2066e97dec4d75a86e72155048de14, 7eecdaa8d1dc4a9988492494fbb17f1d, ae72da07d97b4d9b9e0d246ff83901ed; no voters: I0507 06:31:03.959733 4564 raft_consensus.cc:2640] T 143439bb7e50454aab47fae4ed3ad4ae P 7eecdaa8d1dc4a9988492494fbb17f1d [term 3 FOLLOWER]: Leader election won for term 3 I0507 06:31:03.960151 4564 raft_consensus.cc:667] T 143439bb7e50454aab47fae4ed3ad4ae P 7eecdaa8d1dc4a9988492494fbb17f1d [term 3 LEADER]: Becoming Leader. State: Replica: 7eecdaa8d1dc4a9988492494fbb17f1d, State: Running, Role: LEADER I0507 06:31:03.960696 4564 consensus_queue.cc:206] T 143439bb7e50454aab47fae4ed3ad4ae P 7eecdaa8d1dc4a9988492494fbb17f1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae72da07d97b4d9b9e0d246ff83901ed" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 34185 } } peers { permanent_uuid: "7eecdaa8d1dc4a9988492494fbb17f1d" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 33833 } } peers { permanent_uuid: "a49a0f0a6258477d8126ad3c88bf0d44" member_type: VOTER last_known_addr { host: "127.0.57.196" port: 43761 } } peers { permanent_uuid: "6f2066e97dec4d75a86e72155048de14" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 43041 } attrs { } } I0507 06:31:03.963518 3951 catalog_manager.cc:3988] T 143439bb7e50454aab47fae4ed3ad4ae P 7eecdaa8d1dc4a9988492494fbb17f1d reported cstate change: term changed from 2 to 3, leader changed from a49a0f0a6258477d8126ad3c88bf0d44 (127.0.57.196) to 7eecdaa8d1dc4a9988492494fbb17f1d (127.0.57.195). New cstate: current_term: 3 leader_uuid: "7eecdaa8d1dc4a9988492494fbb17f1d" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae72da07d97b4d9b9e0d246ff83901ed" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 34185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7eecdaa8d1dc4a9988492494fbb17f1d" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 33833 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a49a0f0a6258477d8126ad3c88bf0d44" member_type: VOTER last_known_addr { host: "127.0.57.196" port: 43761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6f2066e97dec4d75a86e72155048de14" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 43041 } attrs { } health_report { overall_health: UNKNOWN } } } I0507 06:31:04.004479 4190 raft_consensus.cc:1181] T 143439bb7e50454aab47fae4ed3ad4ae P ae72da07d97b4d9b9e0d246ff83901ed [term 3 FOLLOWER]: Refusing update from remote peer 7eecdaa8d1dc4a9988492494fbb17f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 5. (index mismatch) I0507 06:31:04.005383 4572 consensus_queue.cc:984] T 143439bb7e50454aab47fae4ed3ad4ae P 7eecdaa8d1dc4a9988492494fbb17f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae72da07d97b4d9b9e0d246ff83901ed" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 34185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s I0507 06:31:04.005956 4072 raft_consensus.cc:1181] T 143439bb7e50454aab47fae4ed3ad4ae P 6f2066e97dec4d75a86e72155048de14 [term 3 FOLLOWER]: Refusing update from remote peer 7eecdaa8d1dc4a9988492494fbb17f1d: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch) I0507 06:31:04.006520 4572 consensus_queue.cc:984] T 143439bb7e50454aab47fae4ed3ad4ae P 7eecdaa8d1dc4a9988492494fbb17f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f2066e97dec4d75a86e72155048de14" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 43041 } attrs { }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s I0507 06:31:04.015748 4440 tablet_service.cc:1286] Received RequestConsensusVote() RPC: tablet_id: "143439bb7e50454aab47fae4ed3ad4ae" candidate_uuid: "7eecdaa8d1dc4a9988492494fbb17f1d" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "a49a0f0a6258477d8126ad3c88bf0d44" is_pre_election: true I0507 06:31:04.015801 4558 consensus_queue.cc:984] T 143439bb7e50454aab47fae4ed3ad4ae P a49a0f0a6258477d8126ad3c88bf0d44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae72da07d97b4d9b9e0d246ff83901ed" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 34185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s I0507 06:31:04.016165 4318 raft_consensus.cc:1149] T 143439bb7e50454aab47fae4ed3ad4ae P 7eecdaa8d1dc4a9988492494fbb17f1d [term 3 LEADER]: Rejecting Update request from peer a49a0f0a6258477d8126ad3c88bf0d44 for earlier term 2. Current term is 3. Ops: [] I0507 06:31:04.016499 4438 tablet_service.cc:1286] Received RequestConsensusVote() RPC: tablet_id: "143439bb7e50454aab47fae4ed3ad4ae" candidate_uuid: "7eecdaa8d1dc4a9988492494fbb17f1d" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "a49a0f0a6258477d8126ad3c88bf0d44" I0507 06:31:04.015748 4437 raft_consensus.cc:2881] T 143439bb7e50454aab47fae4ed3ad4ae P a49a0f0a6258477d8126ad3c88bf0d44 [term 2 LEADER]: Stepping down as leader of term 2 I0507 06:31:04.017225 4437 raft_consensus.cc:705] T 143439bb7e50454aab47fae4ed3ad4ae P a49a0f0a6258477d8126ad3c88bf0d44 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: a49a0f0a6258477d8126ad3c88bf0d44, State: Running, Role: LEADER I0507 06:31:04.017516 4437 consensus_queue.cc:229] T 143439bb7e50454aab47fae4ed3ad4ae P a49a0f0a6258477d8126ad3c88bf0d44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae72da07d97b4d9b9e0d246ff83901ed" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 34185 } } peers { permanent_uuid: "7eecdaa8d1dc4a9988492494fbb17f1d" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 33833 } } peers { permanent_uuid: "a49a0f0a6258477d8126ad3c88bf0d44" member_type: VOTER last_known_addr { host: "127.0.57.196" port: 43761 } } peers { permanent_uuid: "6f2066e97dec4d75a86e72155048de14" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 43041 } attrs { } } I0507 06:31:04.018116 4557 consensus_queue.cc:995] T 143439bb7e50454aab47fae4ed3ad4ae P a49a0f0a6258477d8126ad3c88bf0d44 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7eecdaa8d1dc4a9988492494fbb17f1d" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 33833 }, Status: INVALID_TERM, Last received: 2.4, Next index: 5, Last known committed idx: 5, Time since last communication: 0.000s I0507 06:31:04.018394 4557 consensus_queue.cc:1162] T 143439bb7e50454aab47fae4ed3ad4ae P a49a0f0a6258477d8126ad3c88bf0d44 [NON_LEADER]: Peer 7eecdaa8d1dc4a9988492494fbb17f1d log is divergent from this leader: its last log entry 3.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5 I0507 06:31:04.018478 4190 raft_consensus.cc:1126] T 143439bb7e50454aab47fae4ed3ad4ae P ae72da07d97b4d9b9e0d246ff83901ed [term 3 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[2.4-2.4] Dedup: 2.4->[] I0507 06:31:04.018919 4190 raft_consensus.cc:1149] T 143439bb7e50454aab47fae4ed3ad4ae P ae72da07d97b4d9b9e0d246ff83901ed [term 3 FOLLOWER]: Rejecting Update request from peer a49a0f0a6258477d8126ad3c88bf0d44 for earlier term 2. Current term is 3. Ops: [2.4-2.4] I0507 06:31:04.019750 4557 consensus_queue.cc:995] T 143439bb7e50454aab47fae4ed3ad4ae P a49a0f0a6258477d8126ad3c88bf0d44 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae72da07d97b4d9b9e0d246ff83901ed" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 34185 }, Status: INVALID_TERM, Last received: 1.3, Next index: 4, Last known committed idx: 5, Time since last communication: 0.000s I0507 06:31:04.021430 4437 raft_consensus.cc:2886] T 143439bb7e50454aab47fae4ed3ad4ae P a49a0f0a6258477d8126ad3c88bf0d44 [term 2 FOLLOWER]: Advancing to term 3 I0507 06:31:04.023572 4437 raft_consensus.cc:1181] T 143439bb7e50454aab47fae4ed3ad4ae P a49a0f0a6258477d8126ad3c88bf0d44 [term 3 FOLLOWER]: Refusing update from remote peer 7eecdaa8d1dc4a9988492494fbb17f1d: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch) I0507 06:31:04.023744 4440 raft_consensus.cc:2321] T 143439bb7e50454aab47fae4ed3ad4ae P a49a0f0a6258477d8126ad3c88bf0d44 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7eecdaa8d1dc4a9988492494fbb17f1d for term 3 because replica is already servicing an update from a current leader or another vote. I0507 06:31:04.023921 4438 raft_consensus.cc:2321] T 143439bb7e50454aab47fae4ed3ad4ae P a49a0f0a6258477d8126ad3c88bf0d44 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 7eecdaa8d1dc4a9988492494fbb17f1d for term 3 because replica is already servicing an update from a current leader or another vote. I0507 06:31:04.024266 4439 raft_consensus.cc:1149] T 143439bb7e50454aab47fae4ed3ad4ae P a49a0f0a6258477d8126ad3c88bf0d44 [term 3 FOLLOWER]: Rejecting Update request from peer ae72da07d97b4d9b9e0d246ff83901ed for earlier term 1. Current term is 3. Ops: [] I0507 06:31:04.162138 4530 raft_consensus.cc:472] T 143439bb7e50454aab47fae4ed3ad4ae P 6f2066e97dec4d75a86e72155048de14 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7eecdaa8d1dc4a9988492494fbb17f1d) I0507 06:31:04.162497 4530 raft_consensus.cc:494] T 143439bb7e50454aab47fae4ed3ad4ae P 6f2066e97dec4d75a86e72155048de14 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae72da07d97b4d9b9e0d246ff83901ed" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 34185 } } peers { permanent_uuid: "7eecdaa8d1dc4a9988492494fbb17f1d" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 33833 } } peers { permanent_uuid: "a49a0f0a6258477d8126ad3c88bf0d44" member_type: VOTER last_known_addr { host: "127.0.57.196" port: 43761 } } peers { permanent_uuid: "6f2066e97dec4d75a86e72155048de14" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 43041 } attrs { } } I0507 06:31:04.163447 4530 leader_election.cc:296] T 143439bb7e50454aab47fae4ed3ad4ae P 6f2066e97dec4d75a86e72155048de14 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ae72da07d97b4d9b9e0d246ff83901ed (127.0.57.194:34185), 7eecdaa8d1dc4a9988492494fbb17f1d (127.0.57.195:33833), a49a0f0a6258477d8126ad3c88bf0d44 (127.0.57.196:43761) I0507 06:31:04.163826 4517 raft_consensus.cc:472] T 143439bb7e50454aab47fae4ed3ad4ae P ae72da07d97b4d9b9e0d246ff83901ed [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7eecdaa8d1dc4a9988492494fbb17f1d) I0507 06:31:04.164033 4517 raft_consensus.cc:494] T 143439bb7e50454aab47fae4ed3ad4ae P ae72da07d97b4d9b9e0d246ff83901ed [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae72da07d97b4d9b9e0d246ff83901ed" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 34185 } } peers { permanent_uuid: "7eecdaa8d1dc4a9988492494fbb17f1d" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 33833 } } peers { permanent_uuid: "a49a0f0a6258477d8126ad3c88bf0d44" member_type: VOTER last_known_addr { host: "127.0.57.196" port: 43761 } } peers { permanent_uuid: "6f2066e97dec4d75a86e72155048de14" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 43041 } attrs { } } I0507 06:31:04.164299 4190 tablet_service.cc:1286] Received RequestConsensusVote() RPC: tablet_id: "143439bb7e50454aab47fae4ed3ad4ae" candidate_uuid: "6f2066e97dec4d75a86e72155048de14" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ae72da07d97b4d9b9e0d246ff83901ed" is_pre_election: true I0507 06:31:04.164361 4439 tablet_service.cc:1286] Received RequestConsensusVote() RPC: tablet_id: "143439bb7e50454aab47fae4ed3ad4ae" candidate_uuid: "6f2066e97dec4d75a86e72155048de14" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "a49a0f0a6258477d8126ad3c88bf0d44" is_pre_election: true I0507 06:31:04.164505 4190 raft_consensus.cc:2346] T 143439bb7e50454aab47fae4ed3ad4ae P ae72da07d97b4d9b9e0d246ff83901ed [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6f2066e97dec4d75a86e72155048de14 in term 3. I0507 06:31:04.164691 4439 raft_consensus.cc:2346] T 143439bb7e50454aab47fae4ed3ad4ae P a49a0f0a6258477d8126ad3c88bf0d44 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6f2066e97dec4d75a86e72155048de14 in term 3. I0507 06:31:04.164868 4517 leader_election.cc:296] T 143439bb7e50454aab47fae4ed3ad4ae P ae72da07d97b4d9b9e0d246ff83901ed [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 7eecdaa8d1dc4a9988492494fbb17f1d (127.0.57.195:33833), a49a0f0a6258477d8126ad3c88bf0d44 (127.0.57.196:43761), 6f2066e97dec4d75a86e72155048de14 (127.0.57.193:43041) I0507 06:31:04.165233 4007 leader_election.cc:310] T 143439bb7e50454aab47fae4ed3ad4ae P 6f2066e97dec4d75a86e72155048de14 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 6f2066e97dec4d75a86e72155048de14, a49a0f0a6258477d8126ad3c88bf0d44, ae72da07d97b4d9b9e0d246ff83901ed; no voters: I0507 06:31:04.165570 4530 raft_consensus.cc:2640] T 143439bb7e50454aab47fae4ed3ad4ae P 6f2066e97dec4d75a86e72155048de14 [term 3 FOLLOWER]: Leader pre-election won for term 4 I0507 06:31:04.165647 4530 raft_consensus.cc:472] T 143439bb7e50454aab47fae4ed3ad4ae P 6f2066e97dec4d75a86e72155048de14 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 7eecdaa8d1dc4a9988492494fbb17f1d) I0507 06:31:04.165597 4439 tablet_service.cc:1286] Received RequestConsensusVote() RPC: tablet_id: "143439bb7e50454aab47fae4ed3ad4ae" candidate_uuid: "ae72da07d97b4d9b9e0d246ff83901ed" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "a49a0f0a6258477d8126ad3c88bf0d44" is_pre_election: true I0507 06:31:04.165726 4530 raft_consensus.cc:2886] T 143439bb7e50454aab47fae4ed3ad4ae P 6f2066e97dec4d75a86e72155048de14 [term 3 FOLLOWER]: Advancing to term 4 I0507 06:31:04.165761 4439 raft_consensus.cc:2346] T 143439bb7e50454aab47fae4ed3ad4ae P a49a0f0a6258477d8126ad3c88bf0d44 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae72da07d97b4d9b9e0d246ff83901ed in term 3. I0507 06:31:04.165868 4072 tablet_service.cc:1286] Received RequestConsensusVote() RPC: tablet_id: "143439bb7e50454aab47fae4ed3ad4ae" candidate_uuid: "ae72da07d97b4d9b9e0d246ff83901ed" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6f2066e97dec4d75a86e72155048de14" is_pre_election: true I0507 06:31:04.168200 4530 raft_consensus.cc:494] T 143439bb7e50454aab47fae4ed3ad4ae P 6f2066e97dec4d75a86e72155048de14 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae72da07d97b4d9b9e0d246ff83901ed" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 34185 } } peers { permanent_uuid: "7eecdaa8d1dc4a9988492494fbb17f1d" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 33833 } } peers { permanent_uuid: "a49a0f0a6258477d8126ad3c88bf0d44" member_type: VOTER last_known_addr { host: "127.0.57.196" port: 43761 } } peers { permanent_uuid: "6f2066e97dec4d75a86e72155048de14" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 43041 } attrs { } } I0507 06:31:04.168609 4072 raft_consensus.cc:2276] T 143439bb7e50454aab47fae4ed3ad4ae P 6f2066e97dec4d75a86e72155048de14 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae72da07d97b4d9b9e0d246ff83901ed in current term 4: Already voted for candidate 6f2066e97dec4d75a86e72155048de14 in this term. I0507 06:31:04.169039 4530 leader_election.cc:296] T 143439bb7e50454aab47fae4ed3ad4ae P 6f2066e97dec4d75a86e72155048de14 [CANDIDATE]: Term 4 election: Requested vote from peers ae72da07d97b4d9b9e0d246ff83901ed (127.0.57.194:34185), 7eecdaa8d1dc4a9988492494fbb17f1d (127.0.57.195:33833), a49a0f0a6258477d8126ad3c88bf0d44 (127.0.57.196:43761) I0507 06:31:04.169523 4439 tablet_service.cc:1286] Received RequestConsensusVote() RPC: tablet_id: "143439bb7e50454aab47fae4ed3ad4ae" candidate_uuid: "6f2066e97dec4d75a86e72155048de14" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "a49a0f0a6258477d8126ad3c88bf0d44" I0507 06:31:04.169518 4190 tablet_service.cc:1286] Received RequestConsensusVote() RPC: tablet_id: "143439bb7e50454aab47fae4ed3ad4ae" candidate_uuid: "6f2066e97dec4d75a86e72155048de14" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ae72da07d97b4d9b9e0d246ff83901ed" I0507 06:31:04.169698 4190 raft_consensus.cc:2886] T 143439bb7e50454aab47fae4ed3ad4ae P ae72da07d97b4d9b9e0d246ff83901ed [term 3 FOLLOWER]: Advancing to term 4 I0507 06:31:04.169723 4439 raft_consensus.cc:2886] T 143439bb7e50454aab47fae4ed3ad4ae P a49a0f0a6258477d8126ad3c88bf0d44 [term 3 FOLLOWER]: Advancing to term 4 I0507 06:31:04.172317 4190 raft_consensus.cc:2346] T 143439bb7e50454aab47fae4ed3ad4ae P ae72da07d97b4d9b9e0d246ff83901ed [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6f2066e97dec4d75a86e72155048de14 in term 4. I0507 06:31:04.172466 4439 raft_consensus.cc:2346] T 143439bb7e50454aab47fae4ed3ad4ae P a49a0f0a6258477d8126ad3c88bf0d44 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6f2066e97dec4d75a86e72155048de14 in term 4. I0507 06:31:04.172857 4007 leader_election.cc:310] T 143439bb7e50454aab47fae4ed3ad4ae P 6f2066e97dec4d75a86e72155048de14 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 6f2066e97dec4d75a86e72155048de14, a49a0f0a6258477d8126ad3c88bf0d44, ae72da07d97b4d9b9e0d246ff83901ed; no voters: I0507 06:31:04.173202 4530 raft_consensus.cc:2640] T 143439bb7e50454aab47fae4ed3ad4ae P 6f2066e97dec4d75a86e72155048de14 [term 4 FOLLOWER]: Leader election won for term 4 I0507 06:31:04.173805 4530 raft_consensus.cc:667] T 143439bb7e50454aab47fae4ed3ad4ae P 6f2066e97dec4d75a86e72155048de14 [term 4 LEADER]: Becoming Leader. State: Replica: 6f2066e97dec4d75a86e72155048de14, State: Running, Role: LEADER I0507 06:31:04.174335 4530 consensus_queue.cc:206] T 143439bb7e50454aab47fae4ed3ad4ae P 6f2066e97dec4d75a86e72155048de14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae72da07d97b4d9b9e0d246ff83901ed" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 34185 } } peers { permanent_uuid: "7eecdaa8d1dc4a9988492494fbb17f1d" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 33833 } } peers { permanent_uuid: "a49a0f0a6258477d8126ad3c88bf0d44" member_type: VOTER last_known_addr { host: "127.0.57.196" port: 43761 } } peers { permanent_uuid: "6f2066e97dec4d75a86e72155048de14" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 43041 } attrs { } } I0507 06:31:04.177726 3951 catalog_manager.cc:3988] T 143439bb7e50454aab47fae4ed3ad4ae P 6f2066e97dec4d75a86e72155048de14 reported cstate change: term changed from 3 to 4, leader changed from 7eecdaa8d1dc4a9988492494fbb17f1d (127.0.57.195) to 6f2066e97dec4d75a86e72155048de14 (127.0.57.193). New cstate: current_term: 4 leader_uuid: "6f2066e97dec4d75a86e72155048de14" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae72da07d97b4d9b9e0d246ff83901ed" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 34185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7eecdaa8d1dc4a9988492494fbb17f1d" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 33833 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a49a0f0a6258477d8126ad3c88bf0d44" member_type: VOTER last_known_addr { host: "127.0.57.196" port: 43761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6f2066e97dec4d75a86e72155048de14" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 43041 } attrs { } health_report { overall_health: HEALTHY } } } W0507 06:31:04.232193 4252 leader_election.cc:341] T 143439bb7e50454aab47fae4ed3ad4ae P 7eecdaa8d1dc4a9988492494fbb17f1d [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a49a0f0a6258477d8126ad3c88bf0d44 (127.0.57.196:43761): Timed out: RequestConsensusVote RPC to 127.0.57.196:43761 timed out after 0.151s (SENT) W0507 06:31:04.232772 4252 leader_election.cc:341] T 143439bb7e50454aab47fae4ed3ad4ae P 7eecdaa8d1dc4a9988492494fbb17f1d [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer a49a0f0a6258477d8126ad3c88bf0d44 (127.0.57.196:43761): Timed out: RequestConsensusVote RPC to 127.0.57.196:43761 timed out after 0.162s (SENT) I0507 06:31:04.233261 4318 tablet_service.cc:1286] Received RequestConsensusVote() RPC: tablet_id: "143439bb7e50454aab47fae4ed3ad4ae" candidate_uuid: "6f2066e97dec4d75a86e72155048de14" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7eecdaa8d1dc4a9988492494fbb17f1d" is_pre_election: true I0507 06:31:04.233299 4317 tablet_service.cc:1286] Received RequestConsensusVote() RPC: tablet_id: "143439bb7e50454aab47fae4ed3ad4ae" candidate_uuid: "6f2066e97dec4d75a86e72155048de14" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7eecdaa8d1dc4a9988492494fbb17f1d" I0507 06:31:04.234129 4318 raft_consensus.cc:2307] T 143439bb7e50454aab47fae4ed3ad4ae P 7eecdaa8d1dc4a9988492494fbb17f1d [term 3 LEADER]: Leader pre-election vote request: Denying vote to candidate 6f2066e97dec4d75a86e72155048de14 for term 4 because replica is either leader or believes a valid leader to be alive. I0507 06:31:04.233819 4315 tablet_service.cc:1286] Received RequestConsensusVote() RPC: tablet_id: "143439bb7e50454aab47fae4ed3ad4ae" candidate_uuid: "ae72da07d97b4d9b9e0d246ff83901ed" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7eecdaa8d1dc4a9988492494fbb17f1d" is_pre_election: true I0507 06:31:04.234326 4315 raft_consensus.cc:2307] T 143439bb7e50454aab47fae4ed3ad4ae P 7eecdaa8d1dc4a9988492494fbb17f1d [term 3 LEADER]: Leader pre-election vote request: Denying vote to candidate ae72da07d97b4d9b9e0d246ff83901ed for term 4 because replica is either leader or believes a valid leader to be alive. I0507 06:31:04.234454 4317 raft_consensus.cc:2321] T 143439bb7e50454aab47fae4ed3ad4ae P 7eecdaa8d1dc4a9988492494fbb17f1d [term 3 LEADER]: Leader election vote request: Denying vote to candidate 6f2066e97dec4d75a86e72155048de14 for term 4 because replica is already servicing an update from a current leader or another vote. I0507 06:31:04.234750 4316 raft_consensus.cc:2881] T 143439bb7e50454aab47fae4ed3ad4ae P 7eecdaa8d1dc4a9988492494fbb17f1d [term 3 LEADER]: Stepping down as leader of term 3 I0507 06:31:04.234833 4316 raft_consensus.cc:705] T 143439bb7e50454aab47fae4ed3ad4ae P 7eecdaa8d1dc4a9988492494fbb17f1d [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 7eecdaa8d1dc4a9988492494fbb17f1d, State: Running, Role: LEADER I0507 06:31:04.234972 4190 raft_consensus.cc:1149] T 143439bb7e50454aab47fae4ed3ad4ae P ae72da07d97b4d9b9e0d246ff83901ed [term 4 FOLLOWER]: Rejecting Update request from peer 7eecdaa8d1dc4a9988492494fbb17f1d for earlier term 3. Current term is 4. Ops: [] I0507 06:31:04.235123 4316 consensus_queue.cc:229] T 143439bb7e50454aab47fae4ed3ad4ae P 7eecdaa8d1dc4a9988492494fbb17f1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae72da07d97b4d9b9e0d246ff83901ed" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 34185 } } peers { permanent_uuid: "7eecdaa8d1dc4a9988492494fbb17f1d" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 33833 } } peers { permanent_uuid: "a49a0f0a6258477d8126ad3c88bf0d44" member_type: VOTER last_known_addr { host: "127.0.57.196" port: 43761 } } peers { permanent_uuid: "6f2066e97dec4d75a86e72155048de14" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 43041 } attrs { } } I0507 06:31:04.235568 4129 leader_election.cc:310] T 143439bb7e50454aab47fae4ed3ad4ae P ae72da07d97b4d9b9e0d246ff83901ed [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: a49a0f0a6258477d8126ad3c88bf0d44, ae72da07d97b4d9b9e0d246ff83901ed; no voters: 6f2066e97dec4d75a86e72155048de14, 7eecdaa8d1dc4a9988492494fbb17f1d I0507 06:31:04.235615 4316 raft_consensus.cc:2886] T 143439bb7e50454aab47fae4ed3ad4ae P 7eecdaa8d1dc4a9988492494fbb17f1d [term 3 FOLLOWER]: Advancing to term 4 I0507 06:31:04.235888 4517 raft_consensus.cc:2588] T 143439bb7e50454aab47fae4ed3ad4ae P ae72da07d97b4d9b9e0d246ff83901ed [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority I0507 06:31:04.237054 4190 raft_consensus.cc:1181] T 143439bb7e50454aab47fae4ed3ad4ae P ae72da07d97b4d9b9e0d246ff83901ed [term 4 FOLLOWER]: Refusing update from remote peer 6f2066e97dec4d75a86e72155048de14: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch) I0507 06:31:04.237237 4072 raft_consensus.cc:1149] T 143439bb7e50454aab47fae4ed3ad4ae P 6f2066e97dec4d75a86e72155048de14 [term 4 LEADER]: Rejecting Update request from peer 7eecdaa8d1dc4a9988492494fbb17f1d for earlier term 3. Current term is 4. Ops: [] I0507 06:31:04.238080 4316 raft_consensus.cc:1181] T 143439bb7e50454aab47fae4ed3ad4ae P 7eecdaa8d1dc4a9988492494fbb17f1d [term 4 FOLLOWER]: Refusing update from remote peer 6f2066e97dec4d75a86e72155048de14: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch) I0507 06:31:04.238032 4530 consensus_queue.cc:984] T 143439bb7e50454aab47fae4ed3ad4ae P 6f2066e97dec4d75a86e72155048de14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae72da07d97b4d9b9e0d246ff83901ed" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 34185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s I0507 06:31:04.238369 231 external_mini_cluster.cc:1172] Attempting to check leaks for /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver pid 3987 I0507 06:31:04.238297 4439 raft_consensus.cc:1181] T 143439bb7e50454aab47fae4ed3ad4ae P a49a0f0a6258477d8126ad3c88bf0d44 [term 4 FOLLOWER]: Refusing update from remote peer 6f2066e97dec4d75a86e72155048de14: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 4 index: 6. (index mismatch) I0507 06:31:04.238891 4577 consensus_queue.cc:984] T 143439bb7e50454aab47fae4ed3ad4ae P 6f2066e97dec4d75a86e72155048de14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7eecdaa8d1dc4a9988492494fbb17f1d" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 33833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s I0507 06:31:04.239320 4530 consensus_queue.cc:984] T 143439bb7e50454aab47fae4ed3ad4ae P 6f2066e97dec4d75a86e72155048de14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a49a0f0a6258477d8126ad3c88bf0d44" member_type: VOTER last_known_addr { host: "127.0.57.196" port: 43761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s I0507 06:31:04.240387 4113 generic_service.cc:171] Checking for leaks (request via RPC) I0507 06:31:04.298985 231 external_mini_cluster.cc:1131] Killing /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver with pid 3987 I0507 06:31:04.311373 231 external_mini_cluster.cc:1172] Attempting to check leaks for /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver pid 4120 I0507 06:31:04.312466 4236 generic_service.cc:171] Checking for leaks (request via RPC) I0507 06:31:04.371702 231 external_mini_cluster.cc:1131] Killing /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver with pid 4120 I0507 06:31:04.382580 231 external_mini_cluster.cc:1172] Attempting to check leaks for /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver pid 4242 I0507 06:31:04.383736 4358 generic_service.cc:171] Checking for leaks (request via RPC) I0507 06:31:04.390940 4558 raft_consensus.cc:472] T 143439bb7e50454aab47fae4ed3ad4ae P a49a0f0a6258477d8126ad3c88bf0d44 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 6f2066e97dec4d75a86e72155048de14) I0507 06:31:04.391161 4558 raft_consensus.cc:494] T 143439bb7e50454aab47fae4ed3ad4ae P a49a0f0a6258477d8126ad3c88bf0d44 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae72da07d97b4d9b9e0d246ff83901ed" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 34185 } } peers { permanent_uuid: "7eecdaa8d1dc4a9988492494fbb17f1d" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 33833 } } peers { permanent_uuid: "a49a0f0a6258477d8126ad3c88bf0d44" member_type: VOTER last_known_addr { host: "127.0.57.196" port: 43761 } } peers { permanent_uuid: "6f2066e97dec4d75a86e72155048de14" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 43041 } attrs { } } I0507 06:31:04.392057 4558 leader_election.cc:296] T 143439bb7e50454aab47fae4ed3ad4ae P a49a0f0a6258477d8126ad3c88bf0d44 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ae72da07d97b4d9b9e0d246ff83901ed (127.0.57.194:34185), 7eecdaa8d1dc4a9988492494fbb17f1d (127.0.57.195:33833), 6f2066e97dec4d75a86e72155048de14 (127.0.57.193:43041) W0507 06:31:04.398015 4373 leader_election.cc:341] T 143439bb7e50454aab47fae4ed3ad4ae P a49a0f0a6258477d8126ad3c88bf0d44 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer ae72da07d97b4d9b9e0d246ff83901ed (127.0.57.194:34185): Network error: Client connection negotiation failed: client connection to 127.0.57.194:34185: connect: Connection refused (error 111) W0507 06:31:04.398188 4374 leader_election.cc:341] T 143439bb7e50454aab47fae4ed3ad4ae P a49a0f0a6258477d8126ad3c88bf0d44 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 6f2066e97dec4d75a86e72155048de14 (127.0.57.193:43041): Network error: Client connection negotiation failed: client connection to 127.0.57.193:43041: connect: Connection refused (error 111) I0507 06:31:04.398281 4374 leader_election.cc:310] T 143439bb7e50454aab47fae4ed3ad4ae P a49a0f0a6258477d8126ad3c88bf0d44 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: a49a0f0a6258477d8126ad3c88bf0d44; no voters: 6f2066e97dec4d75a86e72155048de14, ae72da07d97b4d9b9e0d246ff83901ed I0507 06:31:04.398607 4558 raft_consensus.cc:2588] T 143439bb7e50454aab47fae4ed3ad4ae P a49a0f0a6258477d8126ad3c88bf0d44 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority I0507 06:31:04.442373 4575 raft_consensus.cc:472] T 143439bb7e50454aab47fae4ed3ad4ae P 7eecdaa8d1dc4a9988492494fbb17f1d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 6f2066e97dec4d75a86e72155048de14) I0507 06:31:04.441488 4316 tablet_service.cc:1286] Received RequestConsensusVote() RPC: tablet_id: "143439bb7e50454aab47fae4ed3ad4ae" candidate_uuid: "a49a0f0a6258477d8126ad3c88bf0d44" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7eecdaa8d1dc4a9988492494fbb17f1d" is_pre_election: true I0507 06:31:04.442870 4575 raft_consensus.cc:494] T 143439bb7e50454aab47fae4ed3ad4ae P 7eecdaa8d1dc4a9988492494fbb17f1d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae72da07d97b4d9b9e0d246ff83901ed" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 34185 } } peers { permanent_uuid: "7eecdaa8d1dc4a9988492494fbb17f1d" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 33833 } } peers { permanent_uuid: "a49a0f0a6258477d8126ad3c88bf0d44" member_type: VOTER last_known_addr { host: "127.0.57.196" port: 43761 } } peers { permanent_uuid: "6f2066e97dec4d75a86e72155048de14" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 43041 } attrs { } } I0507 06:31:04.443327 4316 raft_consensus.cc:2346] T 143439bb7e50454aab47fae4ed3ad4ae P 7eecdaa8d1dc4a9988492494fbb17f1d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a49a0f0a6258477d8126ad3c88bf0d44 in term 4. I0507 06:31:04.443497 231 external_mini_cluster.cc:1131] Killing /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver with pid 4242 W0507 06:31:04.455404 4373 leader_election.cc:341] T 143439bb7e50454aab47fae4ed3ad4ae P a49a0f0a6258477d8126ad3c88bf0d44 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 7eecdaa8d1dc4a9988492494fbb17f1d (127.0.57.195:33833): Network error: recv got EOF from 127.0.57.195:33833 (error 108) I0507 06:31:04.455534 231 external_mini_cluster.cc:1172] Attempting to check leaks for /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver pid 4364 I0507 06:31:04.456676 4480 generic_service.cc:171] Checking for leaks (request via RPC) I0507 06:31:04.514178 231 external_mini_cluster.cc:1131] Killing /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver with pid 4364 I0507 06:31:04.526245 231 external_mini_cluster.cc:1172] Attempting to check leaks for /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-master pid 3927 I0507 06:31:04.527364 3983 generic_service.cc:171] Checking for leaks (request via RPC) I0507 06:31:04.615324 231 external_mini_cluster.cc:1131] Killing /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-master with pid 3927 [ OK ] RaftConsensusNonVoterITest.PromotedReplicaCanVote (11466 ms) [ RUN ] RaftConsensusNonVoterITest.CatalogManagerEvictsExcessNonVoter I0507 06:31:04.639575 231 test_util.cc:212] Using random seed: 1551998006 I0507 06:31:04.641186 231 ts_itest-base.cc:112] Starting cluster with: I0507 06:31:04.641240 231 ts_itest-base.cc:113] -------------- I0507 06:31:04.641263 231 ts_itest-base.cc:114] 4 tablet servers I0507 06:31:04.641312 231 ts_itest-base.cc:115] 3 replicas per TS I0507 06:31:04.641338 231 ts_itest-base.cc:116] -------------- I0507 06:31:04.642599 231 external_mini_cluster.cc:933] Running /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-master /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-master --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.CatalogManagerEvictsExcessNonVoter.1557210500371863-231/raft_consensus-itest-cluster/master-0/wal --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.CatalogManagerEvictsExcessNonVoter.1557210500371863-231/raft_consensus-itest-cluster/master-0/data --block_manager=log --webserver_interface=localhost --ipki_ca_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.0.57.254:35735 --webserver_interface=127.0.57.254 --webserver_port=0 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.CatalogManagerEvictsExcessNonVoter.1557210500371863-231/raft_consensus-itest-cluster/master-0/logs --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.CatalogManagerEvictsExcessNonVoter.1557210500371863-231/raft_consensus-itest-cluster/master-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=true --catalog_manager_evict_excess_replicas=false --rpc_reuseport=true with env {} WARNING: Logging before InitGoogleLogging() is written to STDERR W0507 06:31:04.850618 4592 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0507 06:31:04.850854 4592 flags.cc:404] Enabled unsafe flag: --never_fsync=true W0507 06:31:04.853374 4592 flags.cc:404] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true W0507 06:31:04.853436 4592 flags.cc:404] Enabled experimental flag: --ipki_ca_key_size=1024 W0507 06:31:04.853467 4592 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W0507 06:31:04.853497 4592 flags.cc:404] Enabled experimental flag: --tsk_num_rsa_bits=512 W0507 06:31:04.853538 4592 flags.cc:404] Enabled experimental flag: --rpc_reuseport=true I0507 06:31:04.858500 4592 master_main.cc:73] Master server non-default flags: --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.CatalogManagerEvictsExcessNonVoter.1557210500371863-231/raft_consensus-itest-cluster/master-0/data --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.CatalogManagerEvictsExcessNonVoter.1557210500371863-231/raft_consensus-itest-cluster/master-0/wal --catalog_manager_evict_excess_replicas=false --ipki_ca_key_size=1024 --ipki_server_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.0.57.254:35735 --rpc_reuseport=true --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.CatalogManagerEvictsExcessNonVoter.1557210500371863-231/raft_consensus-itest-cluster/master-0/data/info.pb --webserver_interface=127.0.57.254 --webserver_port=0 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.CatalogManagerEvictsExcessNonVoter.1557210500371863-231/raft_consensus-itest-cluster/master-0/logs --logbuflevel=-1 --logtostderr=true Master server version: kudu 1.10.0-cdh6.x-SNAPSHOT revision 7d0ee26697926e87af9ef395008799577f6ab43f build type FASTDEBUG built by jenkins at 06 May 2019 22:44:21 PST on quasar-knmpfx-1.vpc.cloudera.com build id 322017 ASAN enabled I0507 06:31:04.865916 4592 system_ntp.cc:202] NTP initialized. Skew: 500ppm Current error: 106516us I0507 06:31:04.867350 4592 fs_manager.cc:263] Metadata directory not provided I0507 06:31:04.867424 4592 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0507 06:31:04.867748 4592 server_base.cc:433] Could not load existing FS layout: Not found: could not find a healthy instance file I0507 06:31:04.867818 4592 server_base.cc:434] Attempting to create new FS layout instead I0507 06:31:04.870726 4592 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.CatalogManagerEvictsExcessNonVoter.1557210500371863-231/raft_consensus-itest-cluster/master-0/data/instance: uuid: "1e022e5c62e64dfca937e8cef14cfb0e" format_stamp: "Formatted at 2019-05-07 06:31:04 on dist-test-slave-dist-test-slave-rqp6" I0507 06:31:04.871310 4592 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.CatalogManagerEvictsExcessNonVoter.1557210500371863-231/raft_consensus-itest-cluster/master-0/wal/instance: uuid: "1e022e5c62e64dfca937e8cef14cfb0e" format_stamp: "Formatted at 2019-05-07 06:31:04 on dist-test-slave-dist-test-slave-rqp6" I0507 06:31:04.878660 4592 fs_manager.cc:503] Time spent creating directory manager: real 0.007s user 0.004s sys 0.004s I0507 06:31:04.879734 4592 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0507 06:31:04.880345 4592 file_cache.cc:466] Constructed file cache lbm with capacity 419430 I0507 06:31:04.883714 4592 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s I0507 06:31:04.883797 4592 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.CatalogManagerEvictsExcessNonVoter.1557210500371863-231/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.CatalogManagerEvictsExcessNonVoter.1557210500371863-231/raft_consensus-itest-cluster/master-0/wal uuid: "1e022e5c62e64dfca937e8cef14cfb0e" format_stamp: "Formatted at 2019-05-07 06:31:04 on dist-test-slave-dist-test-slave-rqp6" I0507 06:31:04.883939 4592 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.CatalogManagerEvictsExcessNonVoter.1557210500371863-231/raft_consensus-itest-cluster/master-0/wal metadata directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.CatalogManagerEvictsExcessNonVoter.1557210500371863-231/raft_consensus-itest-cluster/master-0/wal 1 data directories: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.CatalogManagerEvictsExcessNonVoter.1557210500371863-231/raft_consensus-itest-cluster/master-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0507 06:31:04.965226 4592 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0507 06:31:04.981757 4592 rpc_server.cc:205] RPC server started. Bound to: 127.0.57.254:35735 I0507 06:31:04.983005 4592 webserver.cc:308] Webserver started at http://127.0.57.254:46645/ using document root and password file I0507 06:31:04.983789 4592 server_base.cc:608] Dumped server information to /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.CatalogManagerEvictsExcessNonVoter.1557210500371863-231/raft_consensus-itest-cluster/master-0/data/info.pb I0507 06:31:04.987308 4651 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0507 06:31:04.991968 231 external_mini_cluster.cc:995] Started /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-master as pid 4592 I0507 06:31:04.992836 231 external_mini_cluster.cc:933] Running /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.CatalogManagerEvictsExcessNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-0/wal --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.CatalogManagerEvictsExcessNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data --block_manager=log --rpc_bind_addresses=127.0.57.193:0 --local_ip_for_outbound_sockets=127.0.57.193 --webserver_interface=127.0.57.193 --webserver_port=0 --tserver_master_addrs=127.0.57.254:35735 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.CatalogManagerEvictsExcessNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-0/logs --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.CatalogManagerEvictsExcessNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=true --consensus_rpc_timeout_ms=5000 --follower_unavailable_considered_failed_sec=5 with env {} I0507 06:31:04.998556 4651 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 1e022e5c62e64dfca937e8cef14cfb0e: Bootstrap starting. I0507 06:31:05.001359 4651 tablet_bootstrap.cc:590] T 00000000000000000000000000000000 P 1e022e5c62e64dfca937e8cef14cfb0e: No blocks or log segments found. Creating new log. I0507 06:31:05.002328 4651 log.cc:525] T 00000000000000000000000000000000 P 1e022e5c62e64dfca937e8cef14cfb0e: Log is configured to *not* fsync() on all Append() calls I0507 06:31:05.004312 4651 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 1e022e5c62e64dfca937e8cef14cfb0e: No bootstrap required, opened a new log I0507 06:31:05.010954 4651 raft_consensus.cc:340] T 00000000000000000000000000000000 P 1e022e5c62e64dfca937e8cef14cfb0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e022e5c62e64dfca937e8cef14cfb0e" member_type: VOTER } I0507 06:31:05.011204 4651 raft_consensus.cc:366] T 00000000000000000000000000000000 P 1e022e5c62e64dfca937e8cef14cfb0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0507 06:31:05.011256 4651 raft_consensus.cc:705] T 00000000000000000000000000000000 P 1e022e5c62e64dfca937e8cef14cfb0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e022e5c62e64dfca937e8cef14cfb0e, State: Initialized, Role: FOLLOWER I0507 06:31:05.011860 4651 consensus_queue.cc:229] T 00000000000000000000000000000000 P 1e022e5c62e64dfca937e8cef14cfb0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e022e5c62e64dfca937e8cef14cfb0e" member_type: VOTER } I0507 06:31:05.012094 4651 raft_consensus.cc:380] T 00000000000000000000000000000000 P 1e022e5c62e64dfca937e8cef14cfb0e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0507 06:31:05.012166 4651 raft_consensus.cc:472] T 00000000000000000000000000000000 P 1e022e5c62e64dfca937e8cef14cfb0e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0507 06:31:05.012259 4651 raft_consensus.cc:2886] T 00000000000000000000000000000000 P 1e022e5c62e64dfca937e8cef14cfb0e [term 0 FOLLOWER]: Advancing to term 1 I0507 06:31:05.013921 4651 raft_consensus.cc:494] T 00000000000000000000000000000000 P 1e022e5c62e64dfca937e8cef14cfb0e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e022e5c62e64dfca937e8cef14cfb0e" member_type: VOTER } I0507 06:31:05.014411 4651 leader_election.cc:310] T 00000000000000000000000000000000 P 1e022e5c62e64dfca937e8cef14cfb0e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1e022e5c62e64dfca937e8cef14cfb0e; no voters: I0507 06:31:05.015197 4654 raft_consensus.cc:2640] T 00000000000000000000000000000000 P 1e022e5c62e64dfca937e8cef14cfb0e [term 1 FOLLOWER]: Leader election won for term 1 I0507 06:31:05.015506 4651 leader_election.cc:296] T 00000000000000000000000000000000 P 1e022e5c62e64dfca937e8cef14cfb0e [CANDIDATE]: Term 1 election: Requested vote from peers I0507 06:31:05.016273 4654 raft_consensus.cc:667] T 00000000000000000000000000000000 P 1e022e5c62e64dfca937e8cef14cfb0e [term 1 LEADER]: Becoming Leader. State: Replica: 1e022e5c62e64dfca937e8cef14cfb0e, State: Running, Role: LEADER I0507 06:31:05.016665 4654 consensus_queue.cc:206] T 00000000000000000000000000000000 P 1e022e5c62e64dfca937e8cef14cfb0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e022e5c62e64dfca937e8cef14cfb0e" member_type: VOTER } I0507 06:31:05.020433 4656 sys_catalog.cc:337] T 00000000000000000000000000000000 P 1e022e5c62e64dfca937e8cef14cfb0e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1e022e5c62e64dfca937e8cef14cfb0e. Latest consensus state: current_term: 1 leader_uuid: "1e022e5c62e64dfca937e8cef14cfb0e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e022e5c62e64dfca937e8cef14cfb0e" member_type: VOTER } } I0507 06:31:05.020728 4651 sys_catalog.cc:337] T 00000000000000000000000000000000 P 1e022e5c62e64dfca937e8cef14cfb0e [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 1 leader_uuid: "1e022e5c62e64dfca937e8cef14cfb0e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e022e5c62e64dfca937e8cef14cfb0e" member_type: VOTER } } I0507 06:31:05.020916 4651 sys_catalog.cc:340] T 00000000000000000000000000000000 P 1e022e5c62e64dfca937e8cef14cfb0e [sys.catalog]: This master's current role is: LEADER I0507 06:31:05.021206 4656 sys_catalog.cc:340] T 00000000000000000000000000000000 P 1e022e5c62e64dfca937e8cef14cfb0e [sys.catalog]: This master's current role is: LEADER I0507 06:31:05.021531 4655 sys_catalog.cc:337] T 00000000000000000000000000000000 P 1e022e5c62e64dfca937e8cef14cfb0e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1e022e5c62e64dfca937e8cef14cfb0e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e022e5c62e64dfca937e8cef14cfb0e" member_type: VOTER } } I0507 06:31:05.021782 4655 sys_catalog.cc:340] T 00000000000000000000000000000000 P 1e022e5c62e64dfca937e8cef14cfb0e [sys.catalog]: This master's current role is: LEADER I0507 06:31:05.026044 4651 sys_catalog.cc:424] T 00000000000000000000000000000000 P 1e022e5c62e64dfca937e8cef14cfb0e [sys.catalog]: configured and running, proceeding with master startup. I0507 06:31:05.026496 4660 catalog_manager.cc:1055] Loading table and tablet metadata into memory... I0507 06:31:05.039052 4660 catalog_manager.cc:1066] Initializing Kudu internal certificate authority... I0507 06:31:05.098536 4660 catalog_manager.cc:950] Generated new certificate authority record I0507 06:31:05.100204 4660 catalog_manager.cc:1075] Loading token signing keys... I0507 06:31:05.112743 4660 catalog_manager.cc:4292] T 00000000000000000000000000000000 P 1e022e5c62e64dfca937e8cef14cfb0e: Generated new TSK 0 WARNING: Logging before InitGoogleLogging() is written to STDERR W0507 06:31:05.200356 4652 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0507 06:31:05.200615 4652 flags.cc:404] Enabled unsafe flag: --never_fsync=true W0507 06:31:05.202757 4652 flags.cc:404] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true W0507 06:31:05.202818 4652 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W0507 06:31:05.202919 4652 flags.cc:404] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.57.193 I0507 06:31:05.208794 4652 tablet_server_main.cc:78] Tablet server non-default flags: --consensus_rpc_timeout_ms=5000 --follower_unavailable_considered_failed_sec=5 --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.CatalogManagerEvictsExcessNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.CatalogManagerEvictsExcessNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-0/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.0.57.193:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.CatalogManagerEvictsExcessNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data/info.pb --webserver_interface=127.0.57.193 --webserver_port=0 --tserver_master_addrs=127.0.57.254:35735 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.0.57.193 --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.CatalogManagerEvictsExcessNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-0/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.10.0-cdh6.x-SNAPSHOT revision 7d0ee26697926e87af9ef395008799577f6ab43f build type FASTDEBUG built by jenkins at 06 May 2019 22:44:21 PST on quasar-knmpfx-1.vpc.cloudera.com build id 322017 ASAN enabled I0507 06:31:05.217221 4652 system_ntp.cc:202] NTP initialized. Skew: 500ppm Current error: 107016us I0507 06:31:05.218022 4652 fs_manager.cc:263] Metadata directory not provided I0507 06:31:05.218088 4652 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0507 06:31:05.218387 4652 server_base.cc:433] Could not load existing FS layout: Not found: could not find a healthy instance file I0507 06:31:05.218446 4652 server_base.cc:434] Attempting to create new FS layout instead I0507 06:31:05.221175 4652 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.CatalogManagerEvictsExcessNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data/instance: uuid: "819c384387ef465b9305b696e94d1631" format_stamp: "Formatted at 2019-05-07 06:31:05 on dist-test-slave-dist-test-slave-rqp6" I0507 06:31:05.221768 4652 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.CatalogManagerEvictsExcessNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-0/wal/instance: uuid: "819c384387ef465b9305b696e94d1631" format_stamp: "Formatted at 2019-05-07 06:31:05 on dist-test-slave-dist-test-slave-rqp6" I0507 06:31:05.231535 4652 fs_manager.cc:503] Time spent creating directory manager: real 0.010s user 0.008s sys 0.001s I0507 06:31:05.232812 4652 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0507 06:31:05.233511 4652 file_cache.cc:466] Constructed file cache lbm with capacity 419430 I0507 06:31:05.243189 4652 fs_manager.cc:419] Time spent opening block manager: real 0.008s user 0.000s sys 0.007s I0507 06:31:05.243315 4652 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.CatalogManagerEvictsExcessNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.CatalogManagerEvictsExcessNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-0/wal uuid: "819c384387ef465b9305b696e94d1631" format_stamp: "Formatted at 2019-05-07 06:31:05 on dist-test-slave-dist-test-slave-rqp6" I0507 06:31:05.243523 4652 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.CatalogManagerEvictsExcessNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-0/wal metadata directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.CatalogManagerEvictsExcessNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-0/wal 1 data directories: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.CatalogManagerEvictsExcessNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0507 06:31:05.285353 4652 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0507 06:31:05.289734 4652 ts_tablet_manager.cc:356] Loaded tablet metadata (0 total tablets, 0 live tablets) I0507 06:31:05.289822 4652 ts_tablet_manager.cc:376] Registered 0 tablets I0507 06:31:05.364743 4652 rpc_server.cc:205] RPC server started. Bound to: 127.0.57.193:33615 I0507 06:31:05.366068 4652 webserver.cc:308] Webserver started at http://127.0.57.193:40861/ using document root and password file I0507 06:31:05.366822 4652 server_base.cc:608] Dumped server information to /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.CatalogManagerEvictsExcessNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data/info.pb I0507 06:31:05.371312 231 external_mini_cluster.cc:995] Started /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver as pid 4652 I0507 06:31:05.372006 231 external_mini_cluster.cc:933] Running /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.CatalogManagerEvictsExcessNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-1/wal --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.CatalogManagerEvictsExcessNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-1/data --block_manager=log --rpc_bind_addresses=127.0.57.194:0 --local_ip_for_outbound_sockets=127.0.57.194 --webserver_interface=127.0.57.194 --webserver_port=0 --tserver_master_addrs=127.0.57.254:35735 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.CatalogManagerEvictsExcessNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-1/logs --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.CatalogManagerEvictsExcessNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-1/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=true --consensus_rpc_timeout_ms=5000 --follower_unavailable_considered_failed_sec=5 with env {} I0507 06:31:05.383493 4782 heartbeater.cc:345] Connected to a master server at 127.0.57.254:35735 I0507 06:31:05.383713 4782 heartbeater.cc:426] Registering TS with master... I0507 06:31:05.384291 4782 heartbeater.cc:475] Master 127.0.57.254:35735 requested a full tablet report, sending... I0507 06:31:05.386059 4618 ts_manager.cc:159] Registered new tserver with Master: 819c384387ef465b9305b696e94d1631 (127.0.57.193:33615) I0507 06:31:05.387748 4618 master_service.cc:273] Signed X509 certificate for tserver {username='slave'} at 127.0.57.193:52073 WARNING: Logging before InitGoogleLogging() is written to STDERR W0507 06:31:05.535192 4786 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0507 06:31:05.535466 4786 flags.cc:404] Enabled unsafe flag: --never_fsync=true W0507 06:31:05.537680 4786 flags.cc:404] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true W0507 06:31:05.537748 4786 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W0507 06:31:05.537868 4786 flags.cc:404] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.57.194 I0507 06:31:05.542273 4786 tablet_server_main.cc:78] Tablet server non-default flags: --consensus_rpc_timeout_ms=5000 --follower_unavailable_considered_failed_sec=5 --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.CatalogManagerEvictsExcessNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-1/data --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.CatalogManagerEvictsExcessNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-1/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.0.57.194:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.CatalogManagerEvictsExcessNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-1/data/info.pb --webserver_interface=127.0.57.194 --webserver_port=0 --tserver_master_addrs=127.0.57.254:35735 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.0.57.194 --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.CatalogManagerEvictsExcessNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-1/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.10.0-cdh6.x-SNAPSHOT revision 7d0ee26697926e87af9ef395008799577f6ab43f build type FASTDEBUG built by jenkins at 06 May 2019 22:44:21 PST on quasar-knmpfx-1.vpc.cloudera.com build id 322017 ASAN enabled I0507 06:31:05.550513 4786 system_ntp.cc:202] NTP initialized. Skew: 500ppm Current error: 107016us I0507 06:31:05.551290 4786 fs_manager.cc:263] Metadata directory not provided I0507 06:31:05.551357 4786 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0507 06:31:05.551652 4786 server_base.cc:433] Could not load existing FS layout: Not found: could not find a healthy instance file I0507 06:31:05.551723 4786 server_base.cc:434] Attempting to create new FS layout instead I0507 06:31:05.554412 4786 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.CatalogManagerEvictsExcessNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-1/data/instance: uuid: "4de1f17be0a14fa38ee0d8025b0ec8ea" format_stamp: "Formatted at 2019-05-07 06:31:05 on dist-test-slave-dist-test-slave-rqp6" I0507 06:31:05.555083 4786 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.CatalogManagerEvictsExcessNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-1/wal/instance: uuid: "4de1f17be0a14fa38ee0d8025b0ec8ea" format_stamp: "Formatted at 2019-05-07 06:31:05 on dist-test-slave-dist-test-slave-rqp6" I0507 06:31:05.562342 4786 fs_manager.cc:503] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s I0507 06:31:05.563745 4786 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0507 06:31:05.564455 4786 file_cache.cc:466] Constructed file cache lbm with capacity 419430 I0507 06:31:05.567911 4786 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.000s sys 0.000s I0507 06:31:05.568013 4786 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.CatalogManagerEvictsExcessNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.CatalogManagerEvictsExcessNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-1/wal uuid: "4de1f17be0a14fa38ee0d8025b0ec8ea" format_stamp: "Formatted at 2019-05-07 06:31:05 on dist-test-slave-dist-test-slave-rqp6" I0507 06:31:05.568209 4786 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.CatalogManagerEvictsExcessNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-1/wal metadata directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.CatalogManagerEvictsExcessNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-1/wal 1 data directories: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.CatalogManagerEvictsExcessNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-1/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0507 06:31:05.602260 4786 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0507 06:31:05.604037 4786 ts_tablet_manager.cc:356] Loaded tablet metadata (0 total tablets, 0 live tablets) I0507 06:31:05.604116 4786 ts_tablet_manager.cc:376] Registered 0 tablets I0507 06:31:05.631686 4786 rpc_server.cc:205] RPC server started. Bound to: 127.0.57.194:42351 I0507 06:31:05.632915 4786 webserver.cc:308] Webserver started at http://127.0.57.194:37155/ using document root and password file I0507 06:31:05.633575 4786 server_base.cc:608] Dumped server information to /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.CatalogManagerEvictsExcessNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-1/data/info.pb I0507 06:31:05.639967 231 external_mini_cluster.cc:995] Started /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver as pid 4786 I0507 06:31:05.640921 231 external_mini_cluster.cc:933] Running /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.CatalogManagerEvictsExcessNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-2/wal --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.CatalogManagerEvictsExcessNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-2/data --block_manager=log --rpc_bind_addresses=127.0.57.195:0 --local_ip_for_outbound_sockets=127.0.57.195 --webserver_interface=127.0.57.195 --webserver_port=0 --tserver_master_addrs=127.0.57.254:35735 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.CatalogManagerEvictsExcessNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-2/logs --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.CatalogManagerEvictsExcessNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-2/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=true --consensus_rpc_timeout_ms=5000 --follower_unavailable_considered_failed_sec=5 with env {} I0507 06:31:05.648267 4905 heartbeater.cc:345] Connected to a master server at 127.0.57.254:35735 I0507 06:31:05.648506 4905 heartbeater.cc:426] Registering TS with master... I0507 06:31:05.649201 4905 heartbeater.cc:475] Master 127.0.57.254:35735 requested a full tablet report, sending... I0507 06:31:05.650454 4618 ts_manager.cc:159] Registered new tserver with Master: 4de1f17be0a14fa38ee0d8025b0ec8ea (127.0.57.194:42351) I0507 06:31:05.651412 4618 master_service.cc:273] Signed X509 certificate for tserver {username='slave'} at 127.0.57.194:55529 WARNING: Logging before InitGoogleLogging() is written to STDERR W0507 06:31:05.810663 4908 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0507 06:31:05.810925 4908 flags.cc:404] Enabled unsafe flag: --never_fsync=true W0507 06:31:05.813123 4908 flags.cc:404] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true W0507 06:31:05.813189 4908 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W0507 06:31:05.813287 4908 flags.cc:404] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.57.195 I0507 06:31:05.817378 4908 tablet_server_main.cc:78] Tablet server non-default flags: --consensus_rpc_timeout_ms=5000 --follower_unavailable_considered_failed_sec=5 --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.CatalogManagerEvictsExcessNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-2/data --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.CatalogManagerEvictsExcessNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-2/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.0.57.195:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.CatalogManagerEvictsExcessNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-2/data/info.pb --webserver_interface=127.0.57.195 --webserver_port=0 --tserver_master_addrs=127.0.57.254:35735 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.0.57.195 --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.CatalogManagerEvictsExcessNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-2/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.10.0-cdh6.x-SNAPSHOT revision 7d0ee26697926e87af9ef395008799577f6ab43f build type FASTDEBUG built by jenkins at 06 May 2019 22:44:21 PST on quasar-knmpfx-1.vpc.cloudera.com build id 322017 ASAN enabled I0507 06:31:05.825595 4908 system_ntp.cc:202] NTP initialized. Skew: 500ppm Current error: 107016us I0507 06:31:05.826337 4908 fs_manager.cc:263] Metadata directory not provided I0507 06:31:05.826407 4908 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0507 06:31:05.826714 4908 server_base.cc:433] Could not load existing FS layout: Not found: could not find a healthy instance file I0507 06:31:05.826777 4908 server_base.cc:434] Attempting to create new FS layout instead I0507 06:31:05.829480 4908 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.CatalogManagerEvictsExcessNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-2/data/instance: uuid: "5332b6306171425e96a41aa5c8c7794b" format_stamp: "Formatted at 2019-05-07 06:31:05 on dist-test-slave-dist-test-slave-rqp6" I0507 06:31:05.830083 4908 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.CatalogManagerEvictsExcessNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-2/wal/instance: uuid: "5332b6306171425e96a41aa5c8c7794b" format_stamp: "Formatted at 2019-05-07 06:31:05 on dist-test-slave-dist-test-slave-rqp6" I0507 06:31:05.837040 4908 fs_manager.cc:503] Time spent creating directory manager: real 0.007s user 0.002s sys 0.004s I0507 06:31:05.838177 4908 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0507 06:31:05.838778 4908 file_cache.cc:466] Constructed file cache lbm with capacity 419430 I0507 06:31:05.841917 4908 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s I0507 06:31:05.842005 4908 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.CatalogManagerEvictsExcessNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.CatalogManagerEvictsExcessNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-2/wal uuid: "5332b6306171425e96a41aa5c8c7794b" format_stamp: "Formatted at 2019-05-07 06:31:05 on dist-test-slave-dist-test-slave-rqp6" I0507 06:31:05.842190 4908 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.CatalogManagerEvictsExcessNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-2/wal metadata directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.CatalogManagerEvictsExcessNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-2/wal 1 data directories: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.CatalogManagerEvictsExcessNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-2/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0507 06:31:05.882047 4908 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0507 06:31:05.883925 4908 ts_tablet_manager.cc:356] Loaded tablet metadata (0 total tablets, 0 live tablets) I0507 06:31:05.884006 4908 ts_tablet_manager.cc:376] Registered 0 tablets I0507 06:31:05.909997 4908 rpc_server.cc:205] RPC server started. Bound to: 127.0.57.195:45881 I0507 06:31:05.911403 4908 webserver.cc:308] Webserver started at http://127.0.57.195:43105/ using document root and password file I0507 06:31:05.912176 4908 server_base.cc:608] Dumped server information to /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.CatalogManagerEvictsExcessNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-2/data/info.pb I0507 06:31:05.919368 231 external_mini_cluster.cc:995] Started /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver as pid 4908 I0507 06:31:05.920251 231 external_mini_cluster.cc:933] Running /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.CatalogManagerEvictsExcessNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-3/wal --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.CatalogManagerEvictsExcessNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-3/data --block_manager=log --rpc_bind_addresses=127.0.57.196:0 --local_ip_for_outbound_sockets=127.0.57.196 --webserver_interface=127.0.57.196 --webserver_port=0 --tserver_master_addrs=127.0.57.254:35735 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.CatalogManagerEvictsExcessNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-3/logs --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.CatalogManagerEvictsExcessNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-3/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=true --consensus_rpc_timeout_ms=5000 --follower_unavailable_considered_failed_sec=5 with env {} I0507 06:31:05.926050 5027 heartbeater.cc:345] Connected to a master server at 127.0.57.254:35735 I0507 06:31:05.926295 5027 heartbeater.cc:426] Registering TS with master... I0507 06:31:05.926887 5027 heartbeater.cc:475] Master 127.0.57.254:35735 requested a full tablet report, sending... I0507 06:31:05.928122 4618 ts_manager.cc:159] Registered new tserver with Master: 5332b6306171425e96a41aa5c8c7794b (127.0.57.195:45881) I0507 06:31:05.929141 4618 master_service.cc:273] Signed X509 certificate for tserver {username='slave'} at 127.0.57.195:37997 WARNING: Logging before InitGoogleLogging() is written to STDERR W0507 06:31:06.092103 5030 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0507 06:31:06.092339 5030 flags.cc:404] Enabled unsafe flag: --never_fsync=true W0507 06:31:06.094453 5030 flags.cc:404] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true W0507 06:31:06.094514 5030 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W0507 06:31:06.094625 5030 flags.cc:404] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.57.196 I0507 06:31:06.098642 5030 tablet_server_main.cc:78] Tablet server non-default flags: --consensus_rpc_timeout_ms=5000 --follower_unavailable_considered_failed_sec=5 --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.CatalogManagerEvictsExcessNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-3/data --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.CatalogManagerEvictsExcessNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-3/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.0.57.196:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.CatalogManagerEvictsExcessNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-3/data/info.pb --webserver_interface=127.0.57.196 --webserver_port=0 --tserver_master_addrs=127.0.57.254:35735 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.0.57.196 --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.CatalogManagerEvictsExcessNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-3/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.10.0-cdh6.x-SNAPSHOT revision 7d0ee26697926e87af9ef395008799577f6ab43f build type FASTDEBUG built by jenkins at 06 May 2019 22:44:21 PST on quasar-knmpfx-1.vpc.cloudera.com build id 322017 ASAN enabled I0507 06:31:06.106856 5030 system_ntp.cc:202] NTP initialized. Skew: 500ppm Current error: 107516us I0507 06:31:06.107596 5030 fs_manager.cc:263] Metadata directory not provided I0507 06:31:06.107658 5030 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0507 06:31:06.107962 5030 server_base.cc:433] Could not load existing FS layout: Not found: could not find a healthy instance file I0507 06:31:06.108034 5030 server_base.cc:434] Attempting to create new FS layout instead I0507 06:31:06.110791 5030 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.CatalogManagerEvictsExcessNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-3/data/instance: uuid: "80a0c4bfe149430fae00cc1445886f54" format_stamp: "Formatted at 2019-05-07 06:31:06 on dist-test-slave-dist-test-slave-rqp6" I0507 06:31:06.111414 5030 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.CatalogManagerEvictsExcessNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-3/wal/instance: uuid: "80a0c4bfe149430fae00cc1445886f54" format_stamp: "Formatted at 2019-05-07 06:31:06 on dist-test-slave-dist-test-slave-rqp6" I0507 06:31:06.118393 5030 fs_manager.cc:503] Time spent creating directory manager: real 0.007s user 0.007s sys 0.001s I0507 06:31:06.119490 5030 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0507 06:31:06.120120 5030 file_cache.cc:466] Constructed file cache lbm with capacity 419430 I0507 06:31:06.123368 5030 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s I0507 06:31:06.123448 5030 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.CatalogManagerEvictsExcessNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.CatalogManagerEvictsExcessNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-3/wal uuid: "80a0c4bfe149430fae00cc1445886f54" format_stamp: "Formatted at 2019-05-07 06:31:06 on dist-test-slave-dist-test-slave-rqp6" I0507 06:31:06.123610 5030 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.CatalogManagerEvictsExcessNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-3/wal metadata directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.CatalogManagerEvictsExcessNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-3/wal 1 data directories: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.CatalogManagerEvictsExcessNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-3/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0507 06:31:06.156452 5030 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0507 06:31:06.158413 5030 ts_tablet_manager.cc:356] Loaded tablet metadata (0 total tablets, 0 live tablets) I0507 06:31:06.158495 5030 ts_tablet_manager.cc:376] Registered 0 tablets I0507 06:31:06.185550 5030 rpc_server.cc:205] RPC server started. Bound to: 127.0.57.196:36083 I0507 06:31:06.186918 5030 webserver.cc:308] Webserver started at http://127.0.57.196:34815/ using document root and password file I0507 06:31:06.187670 5030 server_base.cc:608] Dumped server information to /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.CatalogManagerEvictsExcessNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-3/data/info.pb I0507 06:31:06.196552 231 external_mini_cluster.cc:995] Started /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver as pid 5030 I0507 06:31:06.202456 5149 heartbeater.cc:345] Connected to a master server at 127.0.57.254:35735 I0507 06:31:06.202658 5149 heartbeater.cc:426] Registering TS with master... I0507 06:31:06.203199 5149 heartbeater.cc:475] Master 127.0.57.254:35735 requested a full tablet report, sending... I0507 06:31:06.204361 4618 ts_manager.cc:159] Registered new tserver with Master: 80a0c4bfe149430fae00cc1445886f54 (127.0.57.196:36083) I0507 06:31:06.205533 4618 master_service.cc:273] Signed X509 certificate for tserver {username='slave'} at 127.0.57.196:60647 I0507 06:31:06.209290 231 external_mini_cluster.cc:623] 4 TS(s) registered with all masters I0507 06:31:06.221124 4618 catalog_manager.cc:1415] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36564: name: "TestTable" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 3 split_rows_range_bounds { } partition_schema { range_schema { columns { name: "key" } } } I0507 06:31:06.254463 4842 tablet_service.cc:1016] Processing CreateTablet for tablet 7e32c749ffae418da073ceaf9e67aafb (table=TestTable [id=73ae7e04ec4948ab9bd4ff1630362245]), partition=RANGE (key) PARTITION UNBOUNDED I0507 06:31:06.254863 5086 tablet_service.cc:1016] Processing CreateTablet for tablet 7e32c749ffae418da073ceaf9e67aafb (table=TestTable [id=73ae7e04ec4948ab9bd4ff1630362245]), partition=RANGE (key) PARTITION UNBOUNDED I0507 06:31:06.255477 4842 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet 7e32c749ffae418da073ceaf9e67aafb. 1 dirs total, 0 dirs full, 0 dirs failed I0507 06:31:06.255609 5086 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet 7e32c749ffae418da073ceaf9e67aafb. 1 dirs total, 0 dirs full, 0 dirs failed I0507 06:31:06.265086 5169 tablet_bootstrap.cc:439] T 7e32c749ffae418da073ceaf9e67aafb P 80a0c4bfe149430fae00cc1445886f54: Bootstrap starting. I0507 06:31:06.266726 5170 tablet_bootstrap.cc:439] T 7e32c749ffae418da073ceaf9e67aafb P 4de1f17be0a14fa38ee0d8025b0ec8ea: Bootstrap starting. I0507 06:31:06.267830 5169 tablet_bootstrap.cc:590] T 7e32c749ffae418da073ceaf9e67aafb P 80a0c4bfe149430fae00cc1445886f54: No blocks or log segments found. Creating new log. I0507 06:31:06.268497 4964 tablet_service.cc:1016] Processing CreateTablet for tablet 7e32c749ffae418da073ceaf9e67aafb (table=TestTable [id=73ae7e04ec4948ab9bd4ff1630362245]), partition=RANGE (key) PARTITION UNBOUNDED I0507 06:31:06.269418 4964 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet 7e32c749ffae418da073ceaf9e67aafb. 1 dirs total, 0 dirs full, 0 dirs failed I0507 06:31:06.269419 5170 tablet_bootstrap.cc:590] T 7e32c749ffae418da073ceaf9e67aafb P 4de1f17be0a14fa38ee0d8025b0ec8ea: No blocks or log segments found. Creating new log. I0507 06:31:06.269578 5169 log.cc:525] T 7e32c749ffae418da073ceaf9e67aafb P 80a0c4bfe149430fae00cc1445886f54: Log is configured to *not* fsync() on all Append() calls I0507 06:31:06.270331 5170 log.cc:525] T 7e32c749ffae418da073ceaf9e67aafb P 4de1f17be0a14fa38ee0d8025b0ec8ea: Log is configured to *not* fsync() on all Append() calls I0507 06:31:06.271255 5169 tablet_bootstrap.cc:439] T 7e32c749ffae418da073ceaf9e67aafb P 80a0c4bfe149430fae00cc1445886f54: No bootstrap required, opened a new log I0507 06:31:06.271549 5169 ts_tablet_manager.cc:1053] T 7e32c749ffae418da073ceaf9e67aafb P 80a0c4bfe149430fae00cc1445886f54: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s I0507 06:31:06.271899 5170 tablet_bootstrap.cc:439] T 7e32c749ffae418da073ceaf9e67aafb P 4de1f17be0a14fa38ee0d8025b0ec8ea: No bootstrap required, opened a new log I0507 06:31:06.272157 5170 ts_tablet_manager.cc:1053] T 7e32c749ffae418da073ceaf9e67aafb P 4de1f17be0a14fa38ee0d8025b0ec8ea: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s I0507 06:31:06.280263 5169 raft_consensus.cc:340] T 7e32c749ffae418da073ceaf9e67aafb P 80a0c4bfe149430fae00cc1445886f54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a0c4bfe149430fae00cc1445886f54" member_type: VOTER last_known_addr { host: "127.0.57.196" port: 36083 } } peers { permanent_uuid: "4de1f17be0a14fa38ee0d8025b0ec8ea" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 42351 } } peers { permanent_uuid: "5332b6306171425e96a41aa5c8c7794b" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 45881 } } I0507 06:31:06.280665 5169 raft_consensus.cc:366] T 7e32c749ffae418da073ceaf9e67aafb P 80a0c4bfe149430fae00cc1445886f54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0507 06:31:06.280728 5169 raft_consensus.cc:705] T 7e32c749ffae418da073ceaf9e67aafb P 80a0c4bfe149430fae00cc1445886f54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80a0c4bfe149430fae00cc1445886f54, State: Initialized, Role: FOLLOWER I0507 06:31:06.281349 5169 consensus_queue.cc:229] T 7e32c749ffae418da073ceaf9e67aafb P 80a0c4bfe149430fae00cc1445886f54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a0c4bfe149430fae00cc1445886f54" member_type: VOTER last_known_addr { host: "127.0.57.196" port: 36083 } } peers { permanent_uuid: "4de1f17be0a14fa38ee0d8025b0ec8ea" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 42351 } } peers { permanent_uuid: "5332b6306171425e96a41aa5c8c7794b" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 45881 } } I0507 06:31:06.285389 5173 tablet_bootstrap.cc:439] T 7e32c749ffae418da073ceaf9e67aafb P 5332b6306171425e96a41aa5c8c7794b: Bootstrap starting. I0507 06:31:06.288453 5173 tablet_bootstrap.cc:590] T 7e32c749ffae418da073ceaf9e67aafb P 5332b6306171425e96a41aa5c8c7794b: No blocks or log segments found. Creating new log. I0507 06:31:06.280174 5170 raft_consensus.cc:340] T 7e32c749ffae418da073ceaf9e67aafb P 4de1f17be0a14fa38ee0d8025b0ec8ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a0c4bfe149430fae00cc1445886f54" member_type: VOTER last_known_addr { host: "127.0.57.196" port: 36083 } } peers { permanent_uuid: "4de1f17be0a14fa38ee0d8025b0ec8ea" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 42351 } } peers { permanent_uuid: "5332b6306171425e96a41aa5c8c7794b" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 45881 } } I0507 06:31:06.288977 5170 raft_consensus.cc:366] T 7e32c749ffae418da073ceaf9e67aafb P 4de1f17be0a14fa38ee0d8025b0ec8ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0507 06:31:06.289045 5170 raft_consensus.cc:705] T 7e32c749ffae418da073ceaf9e67aafb P 4de1f17be0a14fa38ee0d8025b0ec8ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4de1f17be0a14fa38ee0d8025b0ec8ea, State: Initialized, Role: FOLLOWER I0507 06:31:06.289072 5149 heartbeater.cc:467] Master 127.0.57.254:35735 was elected leader, sending a full tablet report... I0507 06:31:06.289703 5169 ts_tablet_manager.cc:1081] T 7e32c749ffae418da073ceaf9e67aafb P 80a0c4bfe149430fae00cc1445886f54: Time spent starting tablet: real 0.018s user 0.007s sys 0.012s I0507 06:31:06.289690 5170 consensus_queue.cc:229] T 7e32c749ffae418da073ceaf9e67aafb P 4de1f17be0a14fa38ee0d8025b0ec8ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a0c4bfe149430fae00cc1445886f54" member_type: VOTER last_known_addr { host: "127.0.57.196" port: 36083 } } peers { permanent_uuid: "4de1f17be0a14fa38ee0d8025b0ec8ea" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 42351 } } peers { permanent_uuid: "5332b6306171425e96a41aa5c8c7794b" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 45881 } } I0507 06:31:06.290642 4905 heartbeater.cc:467] Master 127.0.57.254:35735 was elected leader, sending a full tablet report... I0507 06:31:06.292340 5173 log.cc:525] T 7e32c749ffae418da073ceaf9e67aafb P 5332b6306171425e96a41aa5c8c7794b: Log is configured to *not* fsync() on all Append() calls I0507 06:31:06.293541 5170 ts_tablet_manager.cc:1081] T 7e32c749ffae418da073ceaf9e67aafb P 4de1f17be0a14fa38ee0d8025b0ec8ea: Time spent starting tablet: real 0.021s user 0.011s sys 0.000s I0507 06:31:06.296514 5173 tablet_bootstrap.cc:439] T 7e32c749ffae418da073ceaf9e67aafb P 5332b6306171425e96a41aa5c8c7794b: No bootstrap required, opened a new log I0507 06:31:06.296890 5173 ts_tablet_manager.cc:1053] T 7e32c749ffae418da073ceaf9e67aafb P 5332b6306171425e96a41aa5c8c7794b: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.002s I0507 06:31:06.302521 5173 raft_consensus.cc:340] T 7e32c749ffae418da073ceaf9e67aafb P 5332b6306171425e96a41aa5c8c7794b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a0c4bfe149430fae00cc1445886f54" member_type: VOTER last_known_addr { host: "127.0.57.196" port: 36083 } } peers { permanent_uuid: "4de1f17be0a14fa38ee0d8025b0ec8ea" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 42351 } } peers { permanent_uuid: "5332b6306171425e96a41aa5c8c7794b" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 45881 } } I0507 06:31:06.302911 5173 raft_consensus.cc:366] T 7e32c749ffae418da073ceaf9e67aafb P 5332b6306171425e96a41aa5c8c7794b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0507 06:31:06.302971 5173 raft_consensus.cc:705] T 7e32c749ffae418da073ceaf9e67aafb P 5332b6306171425e96a41aa5c8c7794b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5332b6306171425e96a41aa5c8c7794b, State: Initialized, Role: FOLLOWER I0507 06:31:06.303411 5173 consensus_queue.cc:229] T 7e32c749ffae418da073ceaf9e67aafb P 5332b6306171425e96a41aa5c8c7794b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a0c4bfe149430fae00cc1445886f54" member_type: VOTER last_known_addr { host: "127.0.57.196" port: 36083 } } peers { permanent_uuid: "4de1f17be0a14fa38ee0d8025b0ec8ea" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 42351 } } peers { permanent_uuid: "5332b6306171425e96a41aa5c8c7794b" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 45881 } } I0507 06:31:06.304244 5027 heartbeater.cc:467] Master 127.0.57.254:35735 was elected leader, sending a full tablet report... I0507 06:31:06.304652 5173 ts_tablet_manager.cc:1081] T 7e32c749ffae418da073ceaf9e67aafb P 5332b6306171425e96a41aa5c8c7794b: Time spent starting tablet: real 0.008s user 0.002s sys 0.006s I0507 06:31:06.372006 5174 raft_consensus.cc:472] T 7e32c749ffae418da073ceaf9e67aafb P 80a0c4bfe149430fae00cc1445886f54 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0507 06:31:06.372247 5174 raft_consensus.cc:494] T 7e32c749ffae418da073ceaf9e67aafb P 80a0c4bfe149430fae00cc1445886f54 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a0c4bfe149430fae00cc1445886f54" member_type: VOTER last_known_addr { host: "127.0.57.196" port: 36083 } } peers { permanent_uuid: "4de1f17be0a14fa38ee0d8025b0ec8ea" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 42351 } } peers { permanent_uuid: "5332b6306171425e96a41aa5c8c7794b" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 45881 } } I0507 06:31:06.373700 5174 leader_election.cc:296] T 7e32c749ffae418da073ceaf9e67aafb P 80a0c4bfe149430fae00cc1445886f54 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4de1f17be0a14fa38ee0d8025b0ec8ea (127.0.57.194:42351), 5332b6306171425e96a41aa5c8c7794b (127.0.57.195:45881) I0507 06:31:06.379786 4862 tablet_service.cc:1286] Received RequestConsensusVote() RPC: tablet_id: "7e32c749ffae418da073ceaf9e67aafb" candidate_uuid: "80a0c4bfe149430fae00cc1445886f54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4de1f17be0a14fa38ee0d8025b0ec8ea" is_pre_election: true I0507 06:31:06.380211 4862 raft_consensus.cc:2346] T 7e32c749ffae418da073ceaf9e67aafb P 4de1f17be0a14fa38ee0d8025b0ec8ea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80a0c4bfe149430fae00cc1445886f54 in term 0. I0507 06:31:06.380309 4984 tablet_service.cc:1286] Received RequestConsensusVote() RPC: tablet_id: "7e32c749ffae418da073ceaf9e67aafb" candidate_uuid: "80a0c4bfe149430fae00cc1445886f54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5332b6306171425e96a41aa5c8c7794b" is_pre_election: true I0507 06:31:06.380647 4984 raft_consensus.cc:2346] T 7e32c749ffae418da073ceaf9e67aafb P 5332b6306171425e96a41aa5c8c7794b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80a0c4bfe149430fae00cc1445886f54 in term 0. I0507 06:31:06.380941 5040 leader_election.cc:310] T 7e32c749ffae418da073ceaf9e67aafb P 80a0c4bfe149430fae00cc1445886f54 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4de1f17be0a14fa38ee0d8025b0ec8ea, 80a0c4bfe149430fae00cc1445886f54; no voters: I0507 06:31:06.381418 5174 raft_consensus.cc:2640] T 7e32c749ffae418da073ceaf9e67aafb P 80a0c4bfe149430fae00cc1445886f54 [term 0 FOLLOWER]: Leader pre-election won for term 1 I0507 06:31:06.381528 5174 raft_consensus.cc:472] T 7e32c749ffae418da073ceaf9e67aafb P 80a0c4bfe149430fae00cc1445886f54 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0507 06:31:06.381588 5174 raft_consensus.cc:2886] T 7e32c749ffae418da073ceaf9e67aafb P 80a0c4bfe149430fae00cc1445886f54 [term 0 FOLLOWER]: Advancing to term 1 I0507 06:31:06.383476 5174 raft_consensus.cc:494] T 7e32c749ffae418da073ceaf9e67aafb P 80a0c4bfe149430fae00cc1445886f54 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a0c4bfe149430fae00cc1445886f54" member_type: VOTER last_known_addr { host: "127.0.57.196" port: 36083 } } peers { permanent_uuid: "4de1f17be0a14fa38ee0d8025b0ec8ea" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 42351 } } peers { permanent_uuid: "5332b6306171425e96a41aa5c8c7794b" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 45881 } } I0507 06:31:06.384052 5174 leader_election.cc:296] T 7e32c749ffae418da073ceaf9e67aafb P 80a0c4bfe149430fae00cc1445886f54 [CANDIDATE]: Term 1 election: Requested vote from peers 4de1f17be0a14fa38ee0d8025b0ec8ea (127.0.57.194:42351), 5332b6306171425e96a41aa5c8c7794b (127.0.57.195:45881) I0507 06:31:06.384563 4862 tablet_service.cc:1286] Received RequestConsensusVote() RPC: tablet_id: "7e32c749ffae418da073ceaf9e67aafb" candidate_uuid: "80a0c4bfe149430fae00cc1445886f54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4de1f17be0a14fa38ee0d8025b0ec8ea" I0507 06:31:06.384649 4984 tablet_service.cc:1286] Received RequestConsensusVote() RPC: tablet_id: "7e32c749ffae418da073ceaf9e67aafb" candidate_uuid: "80a0c4bfe149430fae00cc1445886f54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5332b6306171425e96a41aa5c8c7794b" I0507 06:31:06.384708 4862 raft_consensus.cc:2886] T 7e32c749ffae418da073ceaf9e67aafb P 4de1f17be0a14fa38ee0d8025b0ec8ea [term 0 FOLLOWER]: Advancing to term 1 I0507 06:31:06.384814 4984 raft_consensus.cc:2886] T 7e32c749ffae418da073ceaf9e67aafb P 5332b6306171425e96a41aa5c8c7794b [term 0 FOLLOWER]: Advancing to term 1 W0507 06:31:06.386118 4906 tablet.cc:1786] T 7e32c749ffae418da073ceaf9e67aafb P 4de1f17be0a14fa38ee0d8025b0ec8ea: Can't schedule compaction. Clean time has not been advanced past its initial value. I0507 06:31:06.386777 4862 raft_consensus.cc:2346] T 7e32c749ffae418da073ceaf9e67aafb P 4de1f17be0a14fa38ee0d8025b0ec8ea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80a0c4bfe149430fae00cc1445886f54 in term 1. I0507 06:31:06.387042 4984 raft_consensus.cc:2346] T 7e32c749ffae418da073ceaf9e67aafb P 5332b6306171425e96a41aa5c8c7794b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80a0c4bfe149430fae00cc1445886f54 in term 1. I0507 06:31:06.387229 5040 leader_election.cc:310] T 7e32c749ffae418da073ceaf9e67aafb P 80a0c4bfe149430fae00cc1445886f54 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4de1f17be0a14fa38ee0d8025b0ec8ea, 80a0c4bfe149430fae00cc1445886f54; no voters: I0507 06:31:06.387526 5174 raft_consensus.cc:2640] T 7e32c749ffae418da073ceaf9e67aafb P 80a0c4bfe149430fae00cc1445886f54 [term 1 FOLLOWER]: Leader election won for term 1 I0507 06:31:06.388593 5174 raft_consensus.cc:667] T 7e32c749ffae418da073ceaf9e67aafb P 80a0c4bfe149430fae00cc1445886f54 [term 1 LEADER]: Becoming Leader. State: Replica: 80a0c4bfe149430fae00cc1445886f54, State: Running, Role: LEADER I0507 06:31:06.389008 5174 consensus_queue.cc:206] T 7e32c749ffae418da073ceaf9e67aafb P 80a0c4bfe149430fae00cc1445886f54 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a0c4bfe149430fae00cc1445886f54" member_type: VOTER last_known_addr { host: "127.0.57.196" port: 36083 } } peers { permanent_uuid: "4de1f17be0a14fa38ee0d8025b0ec8ea" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 42351 } } peers { permanent_uuid: "5332b6306171425e96a41aa5c8c7794b" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 45881 } } I0507 06:31:06.390266 4782 heartbeater.cc:467] Master 127.0.57.254:35735 was elected leader, sending a full tablet report... I0507 06:31:06.392863 4618 catalog_manager.cc:3988] T 7e32c749ffae418da073ceaf9e67aafb P 80a0c4bfe149430fae00cc1445886f54 reported cstate change: term changed from 0 to 1, leader changed from to 80a0c4bfe149430fae00cc1445886f54 (127.0.57.196). New cstate: current_term: 1 leader_uuid: "80a0c4bfe149430fae00cc1445886f54" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a0c4bfe149430fae00cc1445886f54" member_type: VOTER last_known_addr { host: "127.0.57.196" port: 36083 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4de1f17be0a14fa38ee0d8025b0ec8ea" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 42351 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5332b6306171425e96a41aa5c8c7794b" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 45881 } health_report { overall_health: UNKNOWN } } } I0507 06:31:06.402459 231 external_mini_cluster.cc:623] 4 TS(s) registered with all masters I0507 06:31:06.403861 231 ts_itest-base.cc:232] Waiting for 0 tablets on tserver 819c384387ef465b9305b696e94d1631 to finish bootstrapping I0507 06:31:06.411945 231 ts_itest-base.cc:232] Waiting for 1 tablets on tserver 4de1f17be0a14fa38ee0d8025b0ec8ea to finish bootstrapping W0507 06:31:06.413558 5028 tablet.cc:1786] T 7e32c749ffae418da073ceaf9e67aafb P 5332b6306171425e96a41aa5c8c7794b: Can't schedule compaction. Clean time has not been advanced past its initial value. I0507 06:31:06.418216 231 ts_itest-base.cc:232] Waiting for 1 tablets on tserver 5332b6306171425e96a41aa5c8c7794b to finish bootstrapping I0507 06:31:06.424166 231 ts_itest-base.cc:232] Waiting for 1 tablets on tserver 80a0c4bfe149430fae00cc1445886f54 to finish bootstrapping W0507 06:31:06.439409 5150 tablet.cc:1786] T 7e32c749ffae418da073ceaf9e67aafb P 80a0c4bfe149430fae00cc1445886f54: Can't schedule compaction. Clean time has not been advanced past its initial value. I0507 06:31:06.781078 4862 raft_consensus.cc:1181] T 7e32c749ffae418da073ceaf9e67aafb P 4de1f17be0a14fa38ee0d8025b0ec8ea [term 1 FOLLOWER]: Refusing update from remote peer 80a0c4bfe149430fae00cc1445886f54: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch) I0507 06:31:06.782135 5179 consensus_queue.cc:984] T 7e32c749ffae418da073ceaf9e67aafb P 80a0c4bfe149430fae00cc1445886f54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4de1f17be0a14fa38ee0d8025b0ec8ea" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 42351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0507 06:31:06.789042 4984 raft_consensus.cc:1181] T 7e32c749ffae418da073ceaf9e67aafb P 5332b6306171425e96a41aa5c8c7794b [term 1 FOLLOWER]: Refusing update from remote peer 80a0c4bfe149430fae00cc1445886f54: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch) I0507 06:31:06.789796 5186 consensus_queue.cc:984] T 7e32c749ffae418da073ceaf9e67aafb P 80a0c4bfe149430fae00cc1445886f54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5332b6306171425e96a41aa5c8c7794b" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 45881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0507 06:31:06.794450 5106 consensus_queue.cc:206] T 7e32c749ffae418da073ceaf9e67aafb P 80a0c4bfe149430fae00cc1445886f54 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "80a0c4bfe149430fae00cc1445886f54" member_type: VOTER last_known_addr { host: "127.0.57.196" port: 36083 } } peers { permanent_uuid: "4de1f17be0a14fa38ee0d8025b0ec8ea" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 42351 } } peers { permanent_uuid: "5332b6306171425e96a41aa5c8c7794b" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 45881 } } peers { permanent_uuid: "819c384387ef465b9305b696e94d1631" member_type: NON_VOTER last_known_addr { host: "127.0.57.193" port: 33615 } attrs { } } I0507 06:31:06.797029 4984 raft_consensus.cc:1181] T 7e32c749ffae418da073ceaf9e67aafb P 5332b6306171425e96a41aa5c8c7794b [term 1 FOLLOWER]: Refusing update from remote peer 80a0c4bfe149430fae00cc1445886f54: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0507 06:31:06.797304 4862 raft_consensus.cc:1181] T 7e32c749ffae418da073ceaf9e67aafb P 4de1f17be0a14fa38ee0d8025b0ec8ea [term 1 FOLLOWER]: Refusing update from remote peer 80a0c4bfe149430fae00cc1445886f54: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0507 06:31:06.797454 5186 consensus_queue.cc:984] T 7e32c749ffae418da073ceaf9e67aafb P 80a0c4bfe149430fae00cc1445886f54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5332b6306171425e96a41aa5c8c7794b" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 45881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s I0507 06:31:06.797857 5174 consensus_queue.cc:984] T 7e32c749ffae418da073ceaf9e67aafb P 80a0c4bfe149430fae00cc1445886f54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4de1f17be0a14fa38ee0d8025b0ec8ea" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 42351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s I0507 06:31:06.799813 5179 raft_consensus.cc:2788] T 7e32c749ffae418da073ceaf9e67aafb P 80a0c4bfe149430fae00cc1445886f54 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 819c384387ef465b9305b696e94d1631 (127.0.57.193) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "80a0c4bfe149430fae00cc1445886f54" member_type: VOTER last_known_addr { host: "127.0.57.196" port: 36083 } } peers { permanent_uuid: "4de1f17be0a14fa38ee0d8025b0ec8ea" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 42351 } } peers { permanent_uuid: "5332b6306171425e96a41aa5c8c7794b" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 45881 } } peers { permanent_uuid: "819c384387ef465b9305b696e94d1631" member_type: NON_VOTER last_known_addr { host: "127.0.57.193" port: 33615 } attrs { } } } I0507 06:31:06.800478 4984 raft_consensus.cc:2788] T 7e32c749ffae418da073ceaf9e67aafb P 5332b6306171425e96a41aa5c8c7794b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 819c384387ef465b9305b696e94d1631 (127.0.57.193) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "80a0c4bfe149430fae00cc1445886f54" member_type: VOTER last_known_addr { host: "127.0.57.196" port: 36083 } } peers { permanent_uuid: "4de1f17be0a14fa38ee0d8025b0ec8ea" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 42351 } } peers { permanent_uuid: "5332b6306171425e96a41aa5c8c7794b" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 45881 } } peers { permanent_uuid: "819c384387ef465b9305b696e94d1631" member_type: NON_VOTER last_known_addr { host: "127.0.57.193" port: 33615 } attrs { } } } I0507 06:31:06.800894 4862 raft_consensus.cc:2788] T 7e32c749ffae418da073ceaf9e67aafb P 4de1f17be0a14fa38ee0d8025b0ec8ea [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 819c384387ef465b9305b696e94d1631 (127.0.57.193) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "80a0c4bfe149430fae00cc1445886f54" member_type: VOTER last_known_addr { host: "127.0.57.196" port: 36083 } } peers { permanent_uuid: "4de1f17be0a14fa38ee0d8025b0ec8ea" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 42351 } } peers { permanent_uuid: "5332b6306171425e96a41aa5c8c7794b" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 45881 } } peers { permanent_uuid: "819c384387ef465b9305b696e94d1631" member_type: NON_VOTER last_known_addr { host: "127.0.57.193" port: 33615 } attrs { } } } W0507 06:31:06.803350 5037 consensus_peers.cc:457] T 7e32c749ffae418da073ceaf9e67aafb P 80a0c4bfe149430fae00cc1445886f54 -> Peer 819c384387ef465b9305b696e94d1631 (127.0.57.193:33615): Couldn't send request to peer 819c384387ef465b9305b696e94d1631. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7e32c749ffae418da073ceaf9e67aafb. This is attempt 1: this message will repeat every 5th retry. I0507 06:31:06.804567 4617 catalog_manager.cc:3988] T 7e32c749ffae418da073ceaf9e67aafb P 5332b6306171425e96a41aa5c8c7794b reported cstate change: config changed from index -1 to 2, NON_VOTER 819c384387ef465b9305b696e94d1631 (127.0.57.193) added. New cstate: current_term: 1 leader_uuid: "80a0c4bfe149430fae00cc1445886f54" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "80a0c4bfe149430fae00cc1445886f54" member_type: VOTER last_known_addr { host: "127.0.57.196" port: 36083 } } peers { permanent_uuid: "4de1f17be0a14fa38ee0d8025b0ec8ea" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 42351 } } peers { permanent_uuid: "5332b6306171425e96a41aa5c8c7794b" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 45881 } } peers { permanent_uuid: "819c384387ef465b9305b696e94d1631" member_type: NON_VOTER last_known_addr { host: "127.0.57.193" port: 33615 } attrs { } } } I0507 06:31:06.828507 4648 generic_service.cc:148] {username='slave'} at 127.0.0.1:36562 changed flags via RPC: catalog_manager_evict_excess_replicas from 'false' to 'true' W0507 06:31:12.336845 5037 consensus_peers.cc:435] T 7e32c749ffae418da073ceaf9e67aafb P 80a0c4bfe149430fae00cc1445886f54 -> Peer 819c384387ef465b9305b696e94d1631 (127.0.57.193:33615): Unable to start Tablet Copy on peer: Timed out: StartTabletCopy RPC to 127.0.57.193:33615 timed out after 5.000s (SENT) I0507 06:31:12.432830 5189 consensus_queue.cc:544] T 7e32c749ffae418da073ceaf9e67aafb P 80a0c4bfe149430fae00cc1445886f54 [LEADER]: Leader has been unable to successfully communicate with peer 819c384387ef465b9305b696e94d1631 for more than 5 seconds (5.630s) I0507 06:31:12.435973 4618 catalog_manager.cc:3739] Sending ChangeConfig:REMOVE_PEER on tablet 7e32c749ffae418da073ceaf9e67aafb (attempt 1) I0507 06:31:12.437502 5106 consensus_queue.cc:206] T 7e32c749ffae418da073ceaf9e67aafb P 80a0c4bfe149430fae00cc1445886f54 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "80a0c4bfe149430fae00cc1445886f54" member_type: VOTER last_known_addr { host: "127.0.57.196" port: 36083 } } peers { permanent_uuid: "4de1f17be0a14fa38ee0d8025b0ec8ea" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 42351 } } peers { permanent_uuid: "5332b6306171425e96a41aa5c8c7794b" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 45881 } } I0507 06:31:12.439873 4984 raft_consensus.cc:1181] T 7e32c749ffae418da073ceaf9e67aafb P 5332b6306171425e96a41aa5c8c7794b [term 1 FOLLOWER]: Refusing update from remote peer 80a0c4bfe149430fae00cc1445886f54: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch) I0507 06:31:12.439977 4862 raft_consensus.cc:1181] T 7e32c749ffae418da073ceaf9e67aafb P 4de1f17be0a14fa38ee0d8025b0ec8ea [term 1 FOLLOWER]: Refusing update from remote peer 80a0c4bfe149430fae00cc1445886f54: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch) I0507 06:31:12.440634 5191 consensus_queue.cc:984] T 7e32c749ffae418da073ceaf9e67aafb P 80a0c4bfe149430fae00cc1445886f54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5332b6306171425e96a41aa5c8c7794b" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 45881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s I0507 06:31:12.440915 5190 consensus_queue.cc:984] T 7e32c749ffae418da073ceaf9e67aafb P 80a0c4bfe149430fae00cc1445886f54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4de1f17be0a14fa38ee0d8025b0ec8ea" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 42351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s I0507 06:31:12.444025 5189 raft_consensus.cc:2788] T 7e32c749ffae418da073ceaf9e67aafb P 80a0c4bfe149430fae00cc1445886f54 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER 819c384387ef465b9305b696e94d1631 (127.0.57.193) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "80a0c4bfe149430fae00cc1445886f54" member_type: VOTER last_known_addr { host: "127.0.57.196" port: 36083 } } peers { permanent_uuid: "4de1f17be0a14fa38ee0d8025b0ec8ea" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 42351 } } peers { permanent_uuid: "5332b6306171425e96a41aa5c8c7794b" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 45881 } } } I0507 06:31:12.444634 4984 raft_consensus.cc:2788] T 7e32c749ffae418da073ceaf9e67aafb P 5332b6306171425e96a41aa5c8c7794b [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER 819c384387ef465b9305b696e94d1631 (127.0.57.193) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "80a0c4bfe149430fae00cc1445886f54" member_type: VOTER last_known_addr { host: "127.0.57.196" port: 36083 } } peers { permanent_uuid: "4de1f17be0a14fa38ee0d8025b0ec8ea" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 42351 } } peers { permanent_uuid: "5332b6306171425e96a41aa5c8c7794b" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 45881 } } } I0507 06:31:12.444808 4862 raft_consensus.cc:2788] T 7e32c749ffae418da073ceaf9e67aafb P 4de1f17be0a14fa38ee0d8025b0ec8ea [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER 819c384387ef465b9305b696e94d1631 (127.0.57.193) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "80a0c4bfe149430fae00cc1445886f54" member_type: VOTER last_known_addr { host: "127.0.57.196" port: 36083 } } peers { permanent_uuid: "4de1f17be0a14fa38ee0d8025b0ec8ea" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 42351 } } peers { permanent_uuid: "5332b6306171425e96a41aa5c8c7794b" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 45881 } } } I0507 06:31:12.446959 4600 catalog_manager.cc:3531] ChangeConfig:REMOVE_PEER RPC for tablet 7e32c749ffae418da073ceaf9e67aafb with cas_config_opid_index 2: ChangeConfig:REMOVE_PEER succeeded (attempt 1) I0507 06:31:12.448881 4618 catalog_manager.cc:3988] T 7e32c749ffae418da073ceaf9e67aafb P 80a0c4bfe149430fae00cc1445886f54 reported cstate change: config changed from index 2 to 3, NON_VOTER 819c384387ef465b9305b696e94d1631 (127.0.57.193) evicted. New cstate: current_term: 1 leader_uuid: "80a0c4bfe149430fae00cc1445886f54" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "80a0c4bfe149430fae00cc1445886f54" member_type: VOTER last_known_addr { host: "127.0.57.196" port: 36083 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4de1f17be0a14fa38ee0d8025b0ec8ea" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 42351 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5332b6306171425e96a41aa5c8c7794b" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 45881 } health_report { overall_health: HEALTHY } } } I0507 06:31:12.453223 4618 catalog_manager.cc:3390] Sending DeleteTablet(TABLET_DATA_TOMBSTONED) for tablet 7e32c749ffae418da073ceaf9e67aafb on 819c384387ef465b9305b696e94d1631 (127.0.57.193:33615) (TS 819c384387ef465b9305b696e94d1631 not found in new config with opid_index 3) W0507 06:31:15.585968 5201 negotiation.cc:313] Failed RPC negotiation. Trace: 0507 06:31:12.453869 (+ 0us) reactor.cc:583] Submitting negotiation task for client connection to 127.0.57.193:33615 0507 06:31:12.459921 (+ 6052us) negotiation.cc:98] Waiting for socket to connect 0507 06:31:12.459941 (+ 20us) client_negotiation.cc:167] Beginning negotiation 0507 06:31:12.460033 (+ 92us) client_negotiation.cc:244] Sending NEGOTIATE NegotiatePB request 0507 06:31:15.584981 (+3124948us) negotiation.cc:304] Negotiation complete: Timed out: Client connection negotiation failed: client connection to 127.0.57.193:33615 Metrics: {"client-negotiator.queue_time_us":5912,"thread_start_us":6199,"threads_started":1} W0507 06:31:15.586357 4599 catalog_manager.cc:3132] TS 819c384387ef465b9305b696e94d1631 (127.0.57.193:33615): Delete Tablet RPC failed for tablet 7e32c749ffae418da073ceaf9e67aafb: Timed out: Client connection negotiation failed: client connection to 127.0.57.193:33615 I0507 06:31:15.586513 4599 catalog_manager.cc:3168] Scheduling retry of 7e32c749ffae418da073ceaf9e67aafb Delete Tablet RPC for TS=819c384387ef465b9305b696e94d1631 with a delay of 32 ms (attempt = 1) I0507 06:31:15.619506 4599 catalog_manager.cc:3390] Sending DeleteTablet(TABLET_DATA_TOMBSTONED) for tablet 7e32c749ffae418da073ceaf9e67aafb on 819c384387ef465b9305b696e94d1631 (127.0.57.193:33615) (TS 819c384387ef465b9305b696e94d1631 not found in new config with opid_index 3) I0507 06:31:16.843996 231 external_mini_cluster.cc:1172] Attempting to check leaks for /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver pid 4652 I0507 06:31:16.847035 4779 generic_service.cc:171] Checking for leaks (request via RPC) I0507 06:31:16.897564 231 external_mini_cluster.cc:1131] Killing /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver with pid 4652 W0507 06:31:16.909626 5201 negotiation.cc:313] Failed RPC negotiation. Trace: 0507 06:31:15.619969 (+ 0us) reactor.cc:583] Submitting negotiation task for client connection to 127.0.57.193:33615 0507 06:31:15.620341 (+ 372us) negotiation.cc:98] Waiting for socket to connect 0507 06:31:15.620375 (+ 34us) client_negotiation.cc:167] Beginning negotiation 0507 06:31:15.620581 (+ 206us) client_negotiation.cc:244] Sending NEGOTIATE NegotiatePB request 0507 06:31:16.909423 (+1288842us) negotiation.cc:304] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.57.193:33615: BlockingRecv error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107) Metrics: {"client-negotiator.queue_time_us":145} W0507 06:31:16.909869 4599 catalog_manager.cc:3132] TS 819c384387ef465b9305b696e94d1631 (127.0.57.193:33615): Delete Tablet RPC failed for tablet 7e32c749ffae418da073ceaf9e67aafb: Network error: Client connection negotiation failed: client connection to 127.0.57.193:33615: BlockingRecv error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107) I0507 06:31:16.909983 4599 catalog_manager.cc:3168] Scheduling retry of 7e32c749ffae418da073ceaf9e67aafb Delete Tablet RPC for TS=819c384387ef465b9305b696e94d1631 with a delay of 79 ms (attempt = 2) I0507 06:31:16.910153 231 external_mini_cluster.cc:1172] Attempting to check leaks for /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver pid 4786 I0507 06:31:16.911099 4902 generic_service.cc:171] Checking for leaks (request via RPC) I0507 06:31:16.960213 231 external_mini_cluster.cc:1131] Killing /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver with pid 4786 I0507 06:31:16.972007 231 external_mini_cluster.cc:1172] Attempting to check leaks for /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver pid 4908 I0507 06:31:16.973206 5024 generic_service.cc:171] Checking for leaks (request via RPC) I0507 06:31:16.989506 4599 catalog_manager.cc:3390] Sending DeleteTablet(TABLET_DATA_TOMBSTONED) for tablet 7e32c749ffae418da073ceaf9e67aafb on 819c384387ef465b9305b696e94d1631 (127.0.57.193:33615) (TS 819c384387ef465b9305b696e94d1631 not found in new config with opid_index 3) W0507 06:31:16.990548 4599 catalog_manager.cc:3132] TS 819c384387ef465b9305b696e94d1631 (127.0.57.193:33615): Delete Tablet RPC failed for tablet 7e32c749ffae418da073ceaf9e67aafb: Network error: Client connection negotiation failed: client connection to 127.0.57.193:33615: connect: Connection refused (error 111) I0507 06:31:16.990626 4599 catalog_manager.cc:3168] Scheduling retry of 7e32c749ffae418da073ceaf9e67aafb Delete Tablet RPC for TS=819c384387ef465b9305b696e94d1631 with a delay of 112 ms (attempt = 3) I0507 06:31:17.025205 231 external_mini_cluster.cc:1131] Killing /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver with pid 4908 I0507 06:31:17.037243 231 external_mini_cluster.cc:1172] Attempting to check leaks for /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver pid 5030 I0507 06:31:17.038441 5146 generic_service.cc:171] Checking for leaks (request via RPC) I0507 06:31:17.091260 231 external_mini_cluster.cc:1131] Killing /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver with pid 5030 I0507 06:31:17.103267 231 external_mini_cluster.cc:1172] Attempting to check leaks for /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-master pid 4592 I0507 06:31:17.103426 4599 catalog_manager.cc:3390] Sending DeleteTablet(TABLET_DATA_TOMBSTONED) for tablet 7e32c749ffae418da073ceaf9e67aafb on 819c384387ef465b9305b696e94d1631 (127.0.57.193:33615) (TS 819c384387ef465b9305b696e94d1631 not found in new config with opid_index 3) I0507 06:31:17.104348 4648 generic_service.cc:171] Checking for leaks (request via RPC) W0507 06:31:17.104331 4599 catalog_manager.cc:3132] TS 819c384387ef465b9305b696e94d1631 (127.0.57.193:33615): Delete Tablet RPC failed for tablet 7e32c749ffae418da073ceaf9e67aafb: Network error: Client connection negotiation failed: client connection to 127.0.57.193:33615: connect: Connection refused (error 111) I0507 06:31:17.104488 4599 catalog_manager.cc:3168] Scheduling retry of 7e32c749ffae418da073ceaf9e67aafb Delete Tablet RPC for TS=819c384387ef465b9305b696e94d1631 with a delay of 175 ms (attempt = 4) I0507 06:31:17.196425 231 external_mini_cluster.cc:1131] Killing /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-master with pid 4592 [ OK ] RaftConsensusNonVoterITest.CatalogManagerEvictsExcessNonVoter (12580 ms) [ RUN ] RaftConsensusNonVoterITest.CatalogManagerAddsNonVoter I0507 06:31:17.220105 231 test_util.cc:212] Using random seed: 1564578536 I0507 06:31:17.221884 231 ts_itest-base.cc:112] Starting cluster with: I0507 06:31:17.221940 231 ts_itest-base.cc:113] -------------- I0507 06:31:17.221958 231 ts_itest-base.cc:114] 4 tablet servers I0507 06:31:17.222008 231 ts_itest-base.cc:115] 3 replicas per TS I0507 06:31:17.222025 231 ts_itest-base.cc:116] -------------- I0507 06:31:17.223278 231 external_mini_cluster.cc:933] Running /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-master /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-master --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.CatalogManagerAddsNonVoter.1557210500371863-231/raft_consensus-itest-cluster/master-0/wal --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.CatalogManagerAddsNonVoter.1557210500371863-231/raft_consensus-itest-cluster/master-0/data --block_manager=log --webserver_interface=localhost --ipki_ca_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.0.57.254:37907 --webserver_interface=127.0.57.254 --webserver_port=0 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.CatalogManagerAddsNonVoter.1557210500371863-231/raft_consensus-itest-cluster/master-0/logs --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.CatalogManagerAddsNonVoter.1557210500371863-231/raft_consensus-itest-cluster/master-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=true --catalog_manager_evict_excess_replicas=false --rpc_reuseport=true with env {} WARNING: Logging before InitGoogleLogging() is written to STDERR W0507 06:31:17.434108 5215 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0507 06:31:17.434370 5215 flags.cc:404] Enabled unsafe flag: --never_fsync=true W0507 06:31:17.436866 5215 flags.cc:404] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true W0507 06:31:17.436936 5215 flags.cc:404] Enabled experimental flag: --ipki_ca_key_size=1024 W0507 06:31:17.437000 5215 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W0507 06:31:17.437022 5215 flags.cc:404] Enabled experimental flag: --tsk_num_rsa_bits=512 W0507 06:31:17.437054 5215 flags.cc:404] Enabled experimental flag: --rpc_reuseport=true I0507 06:31:17.441745 5215 master_main.cc:73] Master server non-default flags: --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.CatalogManagerAddsNonVoter.1557210500371863-231/raft_consensus-itest-cluster/master-0/data --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.CatalogManagerAddsNonVoter.1557210500371863-231/raft_consensus-itest-cluster/master-0/wal --catalog_manager_evict_excess_replicas=false --ipki_ca_key_size=1024 --ipki_server_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.0.57.254:37907 --rpc_reuseport=true --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.CatalogManagerAddsNonVoter.1557210500371863-231/raft_consensus-itest-cluster/master-0/data/info.pb --webserver_interface=127.0.57.254 --webserver_port=0 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.CatalogManagerAddsNonVoter.1557210500371863-231/raft_consensus-itest-cluster/master-0/logs --logbuflevel=-1 --logtostderr=true Master server version: kudu 1.10.0-cdh6.x-SNAPSHOT revision 7d0ee26697926e87af9ef395008799577f6ab43f build type FASTDEBUG built by jenkins at 06 May 2019 22:44:21 PST on quasar-knmpfx-1.vpc.cloudera.com build id 322017 ASAN enabled I0507 06:31:17.449549 5215 system_ntp.cc:202] NTP initialized. Skew: 500ppm Current error: 113016us I0507 06:31:17.450963 5215 fs_manager.cc:263] Metadata directory not provided I0507 06:31:17.451028 5215 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0507 06:31:17.451300 5215 server_base.cc:433] Could not load existing FS layout: Not found: could not find a healthy instance file I0507 06:31:17.451370 5215 server_base.cc:434] Attempting to create new FS layout instead I0507 06:31:17.454088 5215 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.CatalogManagerAddsNonVoter.1557210500371863-231/raft_consensus-itest-cluster/master-0/data/instance: uuid: "d9cc5d1f3ea94f6192b22ce366a14f44" format_stamp: "Formatted at 2019-05-07 06:31:17 on dist-test-slave-dist-test-slave-rqp6" I0507 06:31:17.454650 5215 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.CatalogManagerAddsNonVoter.1557210500371863-231/raft_consensus-itest-cluster/master-0/wal/instance: uuid: "d9cc5d1f3ea94f6192b22ce366a14f44" format_stamp: "Formatted at 2019-05-07 06:31:17 on dist-test-slave-dist-test-slave-rqp6" I0507 06:31:17.462127 5215 fs_manager.cc:503] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s I0507 06:31:17.463232 5215 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0507 06:31:17.463881 5215 file_cache.cc:466] Constructed file cache lbm with capacity 419430 I0507 06:31:17.467535 5215 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s I0507 06:31:17.467628 5215 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.CatalogManagerAddsNonVoter.1557210500371863-231/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.CatalogManagerAddsNonVoter.1557210500371863-231/raft_consensus-itest-cluster/master-0/wal uuid: "d9cc5d1f3ea94f6192b22ce366a14f44" format_stamp: "Formatted at 2019-05-07 06:31:17 on dist-test-slave-dist-test-slave-rqp6" I0507 06:31:17.467814 5215 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.CatalogManagerAddsNonVoter.1557210500371863-231/raft_consensus-itest-cluster/master-0/wal metadata directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.CatalogManagerAddsNonVoter.1557210500371863-231/raft_consensus-itest-cluster/master-0/wal 1 data directories: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.CatalogManagerAddsNonVoter.1557210500371863-231/raft_consensus-itest-cluster/master-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0507 06:31:17.497797 5215 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0507 06:31:17.513478 5215 rpc_server.cc:205] RPC server started. Bound to: 127.0.57.254:37907 I0507 06:31:17.514688 5215 webserver.cc:308] Webserver started at http://127.0.57.254:34457/ using document root and password file I0507 06:31:17.515444 5215 server_base.cc:608] Dumped server information to /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.CatalogManagerAddsNonVoter.1557210500371863-231/raft_consensus-itest-cluster/master-0/data/info.pb I0507 06:31:17.518966 5274 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0507 06:31:17.521138 231 external_mini_cluster.cc:995] Started /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-master as pid 5215 I0507 06:31:17.522024 231 external_mini_cluster.cc:933] Running /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.CatalogManagerAddsNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-0/wal --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.CatalogManagerAddsNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data --block_manager=log --rpc_bind_addresses=127.0.57.193:0 --local_ip_for_outbound_sockets=127.0.57.193 --webserver_interface=127.0.57.193 --webserver_port=0 --tserver_master_addrs=127.0.57.254:37907 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.CatalogManagerAddsNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-0/logs --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.CatalogManagerAddsNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=true --follower_unavailable_considered_failed_sec=10 with env {} I0507 06:31:17.534844 5274 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P d9cc5d1f3ea94f6192b22ce366a14f44: Bootstrap starting. I0507 06:31:17.541888 5274 tablet_bootstrap.cc:590] T 00000000000000000000000000000000 P d9cc5d1f3ea94f6192b22ce366a14f44: No blocks or log segments found. Creating new log. I0507 06:31:17.542910 5274 log.cc:525] T 00000000000000000000000000000000 P d9cc5d1f3ea94f6192b22ce366a14f44: Log is configured to *not* fsync() on all Append() calls I0507 06:31:17.550432 5274 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P d9cc5d1f3ea94f6192b22ce366a14f44: No bootstrap required, opened a new log I0507 06:31:17.557901 5274 raft_consensus.cc:340] T 00000000000000000000000000000000 P d9cc5d1f3ea94f6192b22ce366a14f44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9cc5d1f3ea94f6192b22ce366a14f44" member_type: VOTER } I0507 06:31:17.558198 5274 raft_consensus.cc:366] T 00000000000000000000000000000000 P d9cc5d1f3ea94f6192b22ce366a14f44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0507 06:31:17.558265 5274 raft_consensus.cc:705] T 00000000000000000000000000000000 P d9cc5d1f3ea94f6192b22ce366a14f44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9cc5d1f3ea94f6192b22ce366a14f44, State: Initialized, Role: FOLLOWER I0507 06:31:17.558853 5274 consensus_queue.cc:229] T 00000000000000000000000000000000 P d9cc5d1f3ea94f6192b22ce366a14f44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9cc5d1f3ea94f6192b22ce366a14f44" member_type: VOTER } I0507 06:31:17.559036 5274 raft_consensus.cc:380] T 00000000000000000000000000000000 P d9cc5d1f3ea94f6192b22ce366a14f44 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0507 06:31:17.559114 5274 raft_consensus.cc:472] T 00000000000000000000000000000000 P d9cc5d1f3ea94f6192b22ce366a14f44 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0507 06:31:17.559219 5274 raft_consensus.cc:2886] T 00000000000000000000000000000000 P d9cc5d1f3ea94f6192b22ce366a14f44 [term 0 FOLLOWER]: Advancing to term 1 I0507 06:31:17.561098 5274 raft_consensus.cc:494] T 00000000000000000000000000000000 P d9cc5d1f3ea94f6192b22ce366a14f44 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9cc5d1f3ea94f6192b22ce366a14f44" member_type: VOTER } I0507 06:31:17.561569 5274 leader_election.cc:310] T 00000000000000000000000000000000 P d9cc5d1f3ea94f6192b22ce366a14f44 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d9cc5d1f3ea94f6192b22ce366a14f44; no voters: I0507 06:31:17.562441 5274 leader_election.cc:296] T 00000000000000000000000000000000 P d9cc5d1f3ea94f6192b22ce366a14f44 [CANDIDATE]: Term 1 election: Requested vote from peers I0507 06:31:17.562928 5277 raft_consensus.cc:2640] T 00000000000000000000000000000000 P d9cc5d1f3ea94f6192b22ce366a14f44 [term 1 FOLLOWER]: Leader election won for term 1 I0507 06:31:17.564007 5277 raft_consensus.cc:667] T 00000000000000000000000000000000 P d9cc5d1f3ea94f6192b22ce366a14f44 [term 1 LEADER]: Becoming Leader. State: Replica: d9cc5d1f3ea94f6192b22ce366a14f44, State: Running, Role: LEADER I0507 06:31:17.564478 5277 consensus_queue.cc:206] T 00000000000000000000000000000000 P d9cc5d1f3ea94f6192b22ce366a14f44 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9cc5d1f3ea94f6192b22ce366a14f44" member_type: VOTER } I0507 06:31:17.568281 5279 sys_catalog.cc:337] T 00000000000000000000000000000000 P d9cc5d1f3ea94f6192b22ce366a14f44 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d9cc5d1f3ea94f6192b22ce366a14f44. Latest consensus state: current_term: 1 leader_uuid: "d9cc5d1f3ea94f6192b22ce366a14f44" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9cc5d1f3ea94f6192b22ce366a14f44" member_type: VOTER } } I0507 06:31:17.568544 5279 sys_catalog.cc:340] T 00000000000000000000000000000000 P d9cc5d1f3ea94f6192b22ce366a14f44 [sys.catalog]: This master's current role is: LEADER I0507 06:31:17.569195 5278 sys_catalog.cc:337] T 00000000000000000000000000000000 P d9cc5d1f3ea94f6192b22ce366a14f44 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d9cc5d1f3ea94f6192b22ce366a14f44" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9cc5d1f3ea94f6192b22ce366a14f44" member_type: VOTER } } I0507 06:31:17.569767 5274 sys_catalog.cc:337] T 00000000000000000000000000000000 P d9cc5d1f3ea94f6192b22ce366a14f44 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 1 leader_uuid: "d9cc5d1f3ea94f6192b22ce366a14f44" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9cc5d1f3ea94f6192b22ce366a14f44" member_type: VOTER } } I0507 06:31:17.570053 5278 sys_catalog.cc:340] T 00000000000000000000000000000000 P d9cc5d1f3ea94f6192b22ce366a14f44 [sys.catalog]: This master's current role is: LEADER I0507 06:31:17.570098 5274 sys_catalog.cc:340] T 00000000000000000000000000000000 P d9cc5d1f3ea94f6192b22ce366a14f44 [sys.catalog]: This master's current role is: LEADER I0507 06:31:17.571466 5274 sys_catalog.cc:424] T 00000000000000000000000000000000 P d9cc5d1f3ea94f6192b22ce366a14f44 [sys.catalog]: configured and running, proceeding with master startup. I0507 06:31:17.571645 5283 catalog_manager.cc:1055] Loading table and tablet metadata into memory... I0507 06:31:17.572010 5284 catalog_manager.cc:538] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting I0507 06:31:17.576279 5283 catalog_manager.cc:1066] Initializing Kudu internal certificate authority... I0507 06:31:17.602547 5283 catalog_manager.cc:950] Generated new certificate authority record I0507 06:31:17.604138 5283 catalog_manager.cc:1075] Loading token signing keys... I0507 06:31:17.610487 5283 catalog_manager.cc:4292] T 00000000000000000000000000000000 P d9cc5d1f3ea94f6192b22ce366a14f44: Generated new TSK 0 WARNING: Logging before InitGoogleLogging() is written to STDERR W0507 06:31:17.694326 5275 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0507 06:31:17.694543 5275 flags.cc:404] Enabled unsafe flag: --never_fsync=true W0507 06:31:17.696681 5275 flags.cc:404] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true W0507 06:31:17.696750 5275 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W0507 06:31:17.696836 5275 flags.cc:404] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.57.193 I0507 06:31:17.713093 5275 tablet_server_main.cc:78] Tablet server non-default flags: --follower_unavailable_considered_failed_sec=10 --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.CatalogManagerAddsNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.CatalogManagerAddsNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-0/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.0.57.193:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.CatalogManagerAddsNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data/info.pb --webserver_interface=127.0.57.193 --webserver_port=0 --tserver_master_addrs=127.0.57.254:37907 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.0.57.193 --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.CatalogManagerAddsNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-0/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.10.0-cdh6.x-SNAPSHOT revision 7d0ee26697926e87af9ef395008799577f6ab43f build type FASTDEBUG built by jenkins at 06 May 2019 22:44:21 PST on quasar-knmpfx-1.vpc.cloudera.com build id 322017 ASAN enabled I0507 06:31:17.720638 5275 system_ntp.cc:202] NTP initialized. Skew: 500ppm Current error: 113016us I0507 06:31:17.721316 5275 fs_manager.cc:263] Metadata directory not provided I0507 06:31:17.721379 5275 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0507 06:31:17.721629 5275 server_base.cc:433] Could not load existing FS layout: Not found: could not find a healthy instance file I0507 06:31:17.721694 5275 server_base.cc:434] Attempting to create new FS layout instead I0507 06:31:17.724062 5275 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.CatalogManagerAddsNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data/instance: uuid: "3f283c2f720b44328ae90dbb6ede4f2c" format_stamp: "Formatted at 2019-05-07 06:31:17 on dist-test-slave-dist-test-slave-rqp6" I0507 06:31:17.724603 5275 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.CatalogManagerAddsNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-0/wal/instance: uuid: "3f283c2f720b44328ae90dbb6ede4f2c" format_stamp: "Formatted at 2019-05-07 06:31:17 on dist-test-slave-dist-test-slave-rqp6" I0507 06:31:17.738317 5275 fs_manager.cc:503] Time spent creating directory manager: real 0.014s user 0.004s sys 0.008s I0507 06:31:17.739387 5275 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0507 06:31:17.739974 5275 file_cache.cc:466] Constructed file cache lbm with capacity 419430 I0507 06:31:17.743333 5275 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s I0507 06:31:17.743428 5275 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.CatalogManagerAddsNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.CatalogManagerAddsNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-0/wal uuid: "3f283c2f720b44328ae90dbb6ede4f2c" format_stamp: "Formatted at 2019-05-07 06:31:17 on dist-test-slave-dist-test-slave-rqp6" I0507 06:31:17.743592 5275 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.CatalogManagerAddsNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-0/wal metadata directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.CatalogManagerAddsNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-0/wal 1 data directories: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.CatalogManagerAddsNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0507 06:31:17.812994 5275 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0507 06:31:17.817509 5275 ts_tablet_manager.cc:356] Loaded tablet metadata (0 total tablets, 0 live tablets) I0507 06:31:17.817590 5275 ts_tablet_manager.cc:376] Registered 0 tablets I0507 06:31:17.867776 5275 rpc_server.cc:205] RPC server started. Bound to: 127.0.57.193:39837 I0507 06:31:17.869099 5275 webserver.cc:308] Webserver started at http://127.0.57.193:42593/ using document root and password file I0507 06:31:17.869860 5275 server_base.cc:608] Dumped server information to /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.CatalogManagerAddsNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data/info.pb I0507 06:31:17.878875 231 external_mini_cluster.cc:995] Started /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver as pid 5275 I0507 06:31:17.879508 231 external_mini_cluster.cc:933] Running /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.CatalogManagerAddsNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-1/wal --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.CatalogManagerAddsNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-1/data --block_manager=log --rpc_bind_addresses=127.0.57.194:0 --local_ip_for_outbound_sockets=127.0.57.194 --webserver_interface=127.0.57.194 --webserver_port=0 --tserver_master_addrs=127.0.57.254:37907 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.CatalogManagerAddsNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-1/logs --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.CatalogManagerAddsNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-1/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=true --follower_unavailable_considered_failed_sec=10 with env {} I0507 06:31:17.891788 5405 heartbeater.cc:345] Connected to a master server at 127.0.57.254:37907 I0507 06:31:17.891999 5405 heartbeater.cc:426] Registering TS with master... I0507 06:31:17.892544 5405 heartbeater.cc:475] Master 127.0.57.254:37907 requested a full tablet report, sending... I0507 06:31:17.894222 5241 ts_manager.cc:159] Registered new tserver with Master: 3f283c2f720b44328ae90dbb6ede4f2c (127.0.57.193:39837) I0507 06:31:17.895910 5241 master_service.cc:273] Signed X509 certificate for tserver {username='slave'} at 127.0.57.193:57295 WARNING: Logging before InitGoogleLogging() is written to STDERR W0507 06:31:18.033974 5409 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0507 06:31:18.034190 5409 flags.cc:404] Enabled unsafe flag: --never_fsync=true W0507 06:31:18.036234 5409 flags.cc:404] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true W0507 06:31:18.036289 5409 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W0507 06:31:18.036371 5409 flags.cc:404] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.57.194 I0507 06:31:18.040194 5409 tablet_server_main.cc:78] Tablet server non-default flags: --follower_unavailable_considered_failed_sec=10 --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.CatalogManagerAddsNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-1/data --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.CatalogManagerAddsNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-1/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.0.57.194:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.CatalogManagerAddsNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-1/data/info.pb --webserver_interface=127.0.57.194 --webserver_port=0 --tserver_master_addrs=127.0.57.254:37907 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.0.57.194 --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.CatalogManagerAddsNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-1/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.10.0-cdh6.x-SNAPSHOT revision 7d0ee26697926e87af9ef395008799577f6ab43f build type FASTDEBUG built by jenkins at 06 May 2019 22:44:21 PST on quasar-knmpfx-1.vpc.cloudera.com build id 322017 ASAN enabled I0507 06:31:18.047471 5409 system_ntp.cc:202] NTP initialized. Skew: 500ppm Current error: 113516us I0507 06:31:18.048126 5409 fs_manager.cc:263] Metadata directory not provided I0507 06:31:18.048192 5409 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0507 06:31:18.048418 5409 server_base.cc:433] Could not load existing FS layout: Not found: could not find a healthy instance file I0507 06:31:18.048483 5409 server_base.cc:434] Attempting to create new FS layout instead I0507 06:31:18.050844 5409 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.CatalogManagerAddsNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-1/data/instance: uuid: "57b5634a23754ddfbf10759f1def9b8d" format_stamp: "Formatted at 2019-05-07 06:31:18 on dist-test-slave-dist-test-slave-rqp6" I0507 06:31:18.051371 5409 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.CatalogManagerAddsNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-1/wal/instance: uuid: "57b5634a23754ddfbf10759f1def9b8d" format_stamp: "Formatted at 2019-05-07 06:31:18 on dist-test-slave-dist-test-slave-rqp6" I0507 06:31:18.057525 5409 fs_manager.cc:503] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s I0507 06:31:18.058511 5409 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0507 06:31:18.059093 5409 file_cache.cc:466] Constructed file cache lbm with capacity 419430 I0507 06:31:18.061868 5409 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s I0507 06:31:18.061950 5409 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.CatalogManagerAddsNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.CatalogManagerAddsNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-1/wal uuid: "57b5634a23754ddfbf10759f1def9b8d" format_stamp: "Formatted at 2019-05-07 06:31:18 on dist-test-slave-dist-test-slave-rqp6" I0507 06:31:18.062093 5409 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.CatalogManagerAddsNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-1/wal metadata directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.CatalogManagerAddsNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-1/wal 1 data directories: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.CatalogManagerAddsNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-1/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0507 06:31:18.094769 5409 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0507 06:31:18.096328 5409 ts_tablet_manager.cc:356] Loaded tablet metadata (0 total tablets, 0 live tablets) I0507 06:31:18.096407 5409 ts_tablet_manager.cc:376] Registered 0 tablets I0507 06:31:18.123311 5409 rpc_server.cc:205] RPC server started. Bound to: 127.0.57.194:38629 I0507 06:31:18.124660 5409 webserver.cc:308] Webserver started at http://127.0.57.194:35773/ using document root and password file I0507 06:31:18.125489 5409 server_base.cc:608] Dumped server information to /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.CatalogManagerAddsNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-1/data/info.pb I0507 06:31:18.130236 231 external_mini_cluster.cc:995] Started /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver as pid 5409 I0507 06:31:18.131039 231 external_mini_cluster.cc:933] Running /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.CatalogManagerAddsNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-2/wal --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.CatalogManagerAddsNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-2/data --block_manager=log --rpc_bind_addresses=127.0.57.195:0 --local_ip_for_outbound_sockets=127.0.57.195 --webserver_interface=127.0.57.195 --webserver_port=0 --tserver_master_addrs=127.0.57.254:37907 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.CatalogManagerAddsNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-2/logs --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.CatalogManagerAddsNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-2/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=true --follower_unavailable_considered_failed_sec=10 with env {} I0507 06:31:18.139187 5528 heartbeater.cc:345] Connected to a master server at 127.0.57.254:37907 I0507 06:31:18.139400 5528 heartbeater.cc:426] Registering TS with master... I0507 06:31:18.139946 5528 heartbeater.cc:475] Master 127.0.57.254:37907 requested a full tablet report, sending... I0507 06:31:18.141067 5241 ts_manager.cc:159] Registered new tserver with Master: 57b5634a23754ddfbf10759f1def9b8d (127.0.57.194:38629) I0507 06:31:18.141978 5241 master_service.cc:273] Signed X509 certificate for tserver {username='slave'} at 127.0.57.194:58825 WARNING: Logging before InitGoogleLogging() is written to STDERR W0507 06:31:18.300503 5531 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0507 06:31:18.300777 5531 flags.cc:404] Enabled unsafe flag: --never_fsync=true W0507 06:31:18.302848 5531 flags.cc:404] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true W0507 06:31:18.302902 5531 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W0507 06:31:18.303000 5531 flags.cc:404] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.57.195 I0507 06:31:18.307176 5531 tablet_server_main.cc:78] Tablet server non-default flags: --follower_unavailable_considered_failed_sec=10 --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.CatalogManagerAddsNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-2/data --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.CatalogManagerAddsNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-2/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.0.57.195:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.CatalogManagerAddsNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-2/data/info.pb --webserver_interface=127.0.57.195 --webserver_port=0 --tserver_master_addrs=127.0.57.254:37907 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.0.57.195 --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.CatalogManagerAddsNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-2/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.10.0-cdh6.x-SNAPSHOT revision 7d0ee26697926e87af9ef395008799577f6ab43f build type FASTDEBUG built by jenkins at 06 May 2019 22:44:21 PST on quasar-knmpfx-1.vpc.cloudera.com build id 322017 ASAN enabled I0507 06:31:18.315047 5531 system_ntp.cc:202] NTP initialized. Skew: 500ppm Current error: 113516us I0507 06:31:18.315757 5531 fs_manager.cc:263] Metadata directory not provided I0507 06:31:18.315816 5531 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0507 06:31:18.316088 5531 server_base.cc:433] Could not load existing FS layout: Not found: could not find a healthy instance file I0507 06:31:18.316154 5531 server_base.cc:434] Attempting to create new FS layout instead I0507 06:31:18.318619 5531 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.CatalogManagerAddsNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-2/data/instance: uuid: "a4e7f434e44448c1a89a6436e7c5da3a" format_stamp: "Formatted at 2019-05-07 06:31:18 on dist-test-slave-dist-test-slave-rqp6" I0507 06:31:18.319166 5531 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.CatalogManagerAddsNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-2/wal/instance: uuid: "a4e7f434e44448c1a89a6436e7c5da3a" format_stamp: "Formatted at 2019-05-07 06:31:18 on dist-test-slave-dist-test-slave-rqp6" I0507 06:31:18.325992 5531 fs_manager.cc:503] Time spent creating directory manager: real 0.007s user 0.006s sys 0.002s I0507 06:31:18.326982 5531 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0507 06:31:18.327517 5531 file_cache.cc:466] Constructed file cache lbm with capacity 419430 I0507 06:31:18.330935 5531 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s I0507 06:31:18.331025 5531 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.CatalogManagerAddsNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.CatalogManagerAddsNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-2/wal uuid: "a4e7f434e44448c1a89a6436e7c5da3a" format_stamp: "Formatted at 2019-05-07 06:31:18 on dist-test-slave-dist-test-slave-rqp6" I0507 06:31:18.331172 5531 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.CatalogManagerAddsNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-2/wal metadata directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.CatalogManagerAddsNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-2/wal 1 data directories: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.CatalogManagerAddsNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-2/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0507 06:31:18.380918 5531 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0507 06:31:18.382701 5531 ts_tablet_manager.cc:356] Loaded tablet metadata (0 total tablets, 0 live tablets) I0507 06:31:18.382776 5531 ts_tablet_manager.cc:376] Registered 0 tablets I0507 06:31:18.411953 5531 rpc_server.cc:205] RPC server started. Bound to: 127.0.57.195:34093 I0507 06:31:18.413255 5531 webserver.cc:308] Webserver started at http://127.0.57.195:37909/ using document root and password file I0507 06:31:18.413959 5531 server_base.cc:608] Dumped server information to /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.CatalogManagerAddsNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-2/data/info.pb I0507 06:31:18.418650 231 external_mini_cluster.cc:995] Started /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver as pid 5531 I0507 06:31:18.419368 231 external_mini_cluster.cc:933] Running /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.CatalogManagerAddsNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-3/wal --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.CatalogManagerAddsNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-3/data --block_manager=log --rpc_bind_addresses=127.0.57.196:0 --local_ip_for_outbound_sockets=127.0.57.196 --webserver_interface=127.0.57.196 --webserver_port=0 --tserver_master_addrs=127.0.57.254:37907 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.CatalogManagerAddsNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-3/logs --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.CatalogManagerAddsNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-3/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=true --follower_unavailable_considered_failed_sec=10 with env {} I0507 06:31:18.428151 5650 heartbeater.cc:345] Connected to a master server at 127.0.57.254:37907 I0507 06:31:18.428357 5650 heartbeater.cc:426] Registering TS with master... I0507 06:31:18.428930 5650 heartbeater.cc:475] Master 127.0.57.254:37907 requested a full tablet report, sending... I0507 06:31:18.430069 5241 ts_manager.cc:159] Registered new tserver with Master: a4e7f434e44448c1a89a6436e7c5da3a (127.0.57.195:34093) I0507 06:31:18.431061 5241 master_service.cc:273] Signed X509 certificate for tserver {username='slave'} at 127.0.57.195:36975 WARNING: Logging before InitGoogleLogging() is written to STDERR W0507 06:31:18.585005 5653 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0507 06:31:18.585213 5653 flags.cc:404] Enabled unsafe flag: --never_fsync=true W0507 06:31:18.587232 5653 flags.cc:404] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true W0507 06:31:18.587291 5653 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W0507 06:31:18.587378 5653 flags.cc:404] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.57.196 I0507 06:31:18.591442 5653 tablet_server_main.cc:78] Tablet server non-default flags: --follower_unavailable_considered_failed_sec=10 --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.CatalogManagerAddsNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-3/data --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.CatalogManagerAddsNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-3/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.0.57.196:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.CatalogManagerAddsNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-3/data/info.pb --webserver_interface=127.0.57.196 --webserver_port=0 --tserver_master_addrs=127.0.57.254:37907 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.0.57.196 --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.CatalogManagerAddsNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-3/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.10.0-cdh6.x-SNAPSHOT revision 7d0ee26697926e87af9ef395008799577f6ab43f build type FASTDEBUG built by jenkins at 06 May 2019 22:44:21 PST on quasar-knmpfx-1.vpc.cloudera.com build id 322017 ASAN enabled I0507 06:31:18.598953 5653 system_ntp.cc:202] NTP initialized. Skew: 500ppm Current error: 113516us I0507 06:31:18.599609 5653 fs_manager.cc:263] Metadata directory not provided I0507 06:31:18.599680 5653 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0507 06:31:18.599938 5653 server_base.cc:433] Could not load existing FS layout: Not found: could not find a healthy instance file I0507 06:31:18.600000 5653 server_base.cc:434] Attempting to create new FS layout instead I0507 06:31:18.602453 5653 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.CatalogManagerAddsNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-3/data/instance: uuid: "4df733e8c0ed4f95b435c24c1f59c84c" format_stamp: "Formatted at 2019-05-07 06:31:18 on dist-test-slave-dist-test-slave-rqp6" I0507 06:31:18.602994 5653 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.CatalogManagerAddsNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-3/wal/instance: uuid: "4df733e8c0ed4f95b435c24c1f59c84c" format_stamp: "Formatted at 2019-05-07 06:31:18 on dist-test-slave-dist-test-slave-rqp6" I0507 06:31:18.609555 5653 fs_manager.cc:503] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s I0507 06:31:18.610608 5653 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0507 06:31:18.611208 5653 file_cache.cc:466] Constructed file cache lbm with capacity 419430 I0507 06:31:18.614394 5653 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s I0507 06:31:18.614485 5653 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.CatalogManagerAddsNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.CatalogManagerAddsNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-3/wal uuid: "4df733e8c0ed4f95b435c24c1f59c84c" format_stamp: "Formatted at 2019-05-07 06:31:18 on dist-test-slave-dist-test-slave-rqp6" I0507 06:31:18.614648 5653 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.CatalogManagerAddsNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-3/wal metadata directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.CatalogManagerAddsNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-3/wal 1 data directories: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.CatalogManagerAddsNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-3/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0507 06:31:18.660280 5653 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0507 06:31:18.662042 5653 ts_tablet_manager.cc:356] Loaded tablet metadata (0 total tablets, 0 live tablets) I0507 06:31:18.662115 5653 ts_tablet_manager.cc:376] Registered 0 tablets I0507 06:31:18.690598 5653 rpc_server.cc:205] RPC server started. Bound to: 127.0.57.196:40131 I0507 06:31:18.691771 5653 webserver.cc:308] Webserver started at http://127.0.57.196:34233/ using document root and password file I0507 06:31:18.692390 5653 server_base.cc:608] Dumped server information to /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.CatalogManagerAddsNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-3/data/info.pb I0507 06:31:18.698650 231 external_mini_cluster.cc:995] Started /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver as pid 5653 I0507 06:31:18.705446 5772 heartbeater.cc:345] Connected to a master server at 127.0.57.254:37907 I0507 06:31:18.705670 5772 heartbeater.cc:426] Registering TS with master... I0507 06:31:18.706255 5772 heartbeater.cc:475] Master 127.0.57.254:37907 requested a full tablet report, sending... I0507 06:31:18.707403 5241 ts_manager.cc:159] Registered new tserver with Master: 4df733e8c0ed4f95b435c24c1f59c84c (127.0.57.196:40131) I0507 06:31:18.708343 5241 master_service.cc:273] Signed X509 certificate for tserver {username='slave'} at 127.0.57.196:45765 I0507 06:31:18.710080 231 external_mini_cluster.cc:623] 4 TS(s) registered with all masters I0507 06:31:18.725133 5241 catalog_manager.cc:1415] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59512: name: "TestTable" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 3 split_rows_range_bounds { } partition_schema { range_schema { columns { name: "key" } } } I0507 06:31:18.758101 5465 tablet_service.cc:1016] Processing CreateTablet for tablet ea48f25f7b394ecf8f30f2edcc242b0c (table=TestTable [id=951be2c0499f435f94d15606f411000f]), partition=RANGE (key) PARTITION UNBOUNDED I0507 06:31:18.758157 5587 tablet_service.cc:1016] Processing CreateTablet for tablet ea48f25f7b394ecf8f30f2edcc242b0c (table=TestTable [id=951be2c0499f435f94d15606f411000f]), partition=RANGE (key) PARTITION UNBOUNDED I0507 06:31:18.758671 5709 tablet_service.cc:1016] Processing CreateTablet for tablet ea48f25f7b394ecf8f30f2edcc242b0c (table=TestTable [id=951be2c0499f435f94d15606f411000f]), partition=RANGE (key) PARTITION UNBOUNDED I0507 06:31:18.758971 5465 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet ea48f25f7b394ecf8f30f2edcc242b0c. 1 dirs total, 0 dirs full, 0 dirs failed I0507 06:31:18.759059 5587 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet ea48f25f7b394ecf8f30f2edcc242b0c. 1 dirs total, 0 dirs full, 0 dirs failed I0507 06:31:18.759351 5709 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet ea48f25f7b394ecf8f30f2edcc242b0c. 1 dirs total, 0 dirs full, 0 dirs failed I0507 06:31:18.767925 5793 tablet_bootstrap.cc:439] T ea48f25f7b394ecf8f30f2edcc242b0c P 4df733e8c0ed4f95b435c24c1f59c84c: Bootstrap starting. I0507 06:31:18.770660 5793 tablet_bootstrap.cc:590] T ea48f25f7b394ecf8f30f2edcc242b0c P 4df733e8c0ed4f95b435c24c1f59c84c: No blocks or log segments found. Creating new log. I0507 06:31:18.771270 5794 tablet_bootstrap.cc:439] T ea48f25f7b394ecf8f30f2edcc242b0c P a4e7f434e44448c1a89a6436e7c5da3a: Bootstrap starting. I0507 06:31:18.773120 5792 tablet_bootstrap.cc:439] T ea48f25f7b394ecf8f30f2edcc242b0c P 57b5634a23754ddfbf10759f1def9b8d: Bootstrap starting. I0507 06:31:18.775343 5792 tablet_bootstrap.cc:590] T ea48f25f7b394ecf8f30f2edcc242b0c P 57b5634a23754ddfbf10759f1def9b8d: No blocks or log segments found. Creating new log. I0507 06:31:18.776320 5792 log.cc:525] T ea48f25f7b394ecf8f30f2edcc242b0c P 57b5634a23754ddfbf10759f1def9b8d: Log is configured to *not* fsync() on all Append() calls I0507 06:31:18.777407 5793 log.cc:525] T ea48f25f7b394ecf8f30f2edcc242b0c P 4df733e8c0ed4f95b435c24c1f59c84c: Log is configured to *not* fsync() on all Append() calls I0507 06:31:18.779132 5794 tablet_bootstrap.cc:590] T ea48f25f7b394ecf8f30f2edcc242b0c P a4e7f434e44448c1a89a6436e7c5da3a: No blocks or log segments found. Creating new log. I0507 06:31:18.779317 5793 tablet_bootstrap.cc:439] T ea48f25f7b394ecf8f30f2edcc242b0c P 4df733e8c0ed4f95b435c24c1f59c84c: No bootstrap required, opened a new log I0507 06:31:18.779605 5793 ts_tablet_manager.cc:1053] T ea48f25f7b394ecf8f30f2edcc242b0c P 4df733e8c0ed4f95b435c24c1f59c84c: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.000s I0507 06:31:18.779945 5794 log.cc:525] T ea48f25f7b394ecf8f30f2edcc242b0c P a4e7f434e44448c1a89a6436e7c5da3a: Log is configured to *not* fsync() on all Append() calls I0507 06:31:18.781273 5792 tablet_bootstrap.cc:439] T ea48f25f7b394ecf8f30f2edcc242b0c P 57b5634a23754ddfbf10759f1def9b8d: No bootstrap required, opened a new log I0507 06:31:18.781522 5792 ts_tablet_manager.cc:1053] T ea48f25f7b394ecf8f30f2edcc242b0c P 57b5634a23754ddfbf10759f1def9b8d: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.004s I0507 06:31:18.781531 5794 tablet_bootstrap.cc:439] T ea48f25f7b394ecf8f30f2edcc242b0c P a4e7f434e44448c1a89a6436e7c5da3a: No bootstrap required, opened a new log I0507 06:31:18.781776 5794 ts_tablet_manager.cc:1053] T ea48f25f7b394ecf8f30f2edcc242b0c P a4e7f434e44448c1a89a6436e7c5da3a: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.000s I0507 06:31:18.785086 5793 raft_consensus.cc:340] T ea48f25f7b394ecf8f30f2edcc242b0c P 4df733e8c0ed4f95b435c24c1f59c84c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4e7f434e44448c1a89a6436e7c5da3a" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 34093 } } peers { permanent_uuid: "4df733e8c0ed4f95b435c24c1f59c84c" member_type: VOTER last_known_addr { host: "127.0.57.196" port: 40131 } } peers { permanent_uuid: "57b5634a23754ddfbf10759f1def9b8d" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 38629 } } I0507 06:31:18.785655 5793 raft_consensus.cc:366] T ea48f25f7b394ecf8f30f2edcc242b0c P 4df733e8c0ed4f95b435c24c1f59c84c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0507 06:31:18.785764 5793 raft_consensus.cc:705] T ea48f25f7b394ecf8f30f2edcc242b0c P 4df733e8c0ed4f95b435c24c1f59c84c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4df733e8c0ed4f95b435c24c1f59c84c, State: Initialized, Role: FOLLOWER I0507 06:31:18.786314 5793 consensus_queue.cc:229] T ea48f25f7b394ecf8f30f2edcc242b0c P 4df733e8c0ed4f95b435c24c1f59c84c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4e7f434e44448c1a89a6436e7c5da3a" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 34093 } } peers { permanent_uuid: "4df733e8c0ed4f95b435c24c1f59c84c" member_type: VOTER last_known_addr { host: "127.0.57.196" port: 40131 } } peers { permanent_uuid: "57b5634a23754ddfbf10759f1def9b8d" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 38629 } } I0507 06:31:18.787024 5794 raft_consensus.cc:340] T ea48f25f7b394ecf8f30f2edcc242b0c P a4e7f434e44448c1a89a6436e7c5da3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4e7f434e44448c1a89a6436e7c5da3a" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 34093 } } peers { permanent_uuid: "4df733e8c0ed4f95b435c24c1f59c84c" member_type: VOTER last_known_addr { host: "127.0.57.196" port: 40131 } } peers { permanent_uuid: "57b5634a23754ddfbf10759f1def9b8d" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 38629 } } I0507 06:31:18.787307 5794 raft_consensus.cc:366] T ea48f25f7b394ecf8f30f2edcc242b0c P a4e7f434e44448c1a89a6436e7c5da3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0507 06:31:18.787081 5792 raft_consensus.cc:340] T ea48f25f7b394ecf8f30f2edcc242b0c P 57b5634a23754ddfbf10759f1def9b8d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4e7f434e44448c1a89a6436e7c5da3a" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 34093 } } peers { permanent_uuid: "4df733e8c0ed4f95b435c24c1f59c84c" member_type: VOTER last_known_addr { host: "127.0.57.196" port: 40131 } } peers { permanent_uuid: "57b5634a23754ddfbf10759f1def9b8d" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 38629 } } I0507 06:31:18.787376 5794 raft_consensus.cc:705] T ea48f25f7b394ecf8f30f2edcc242b0c P a4e7f434e44448c1a89a6436e7c5da3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4e7f434e44448c1a89a6436e7c5da3a, State: Initialized, Role: FOLLOWER I0507 06:31:18.787407 5792 raft_consensus.cc:366] T ea48f25f7b394ecf8f30f2edcc242b0c P 57b5634a23754ddfbf10759f1def9b8d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0507 06:31:18.787840 5772 heartbeater.cc:467] Master 127.0.57.254:37907 was elected leader, sending a full tablet report... I0507 06:31:18.787833 5792 raft_consensus.cc:705] T ea48f25f7b394ecf8f30f2edcc242b0c P 57b5634a23754ddfbf10759f1def9b8d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57b5634a23754ddfbf10759f1def9b8d, State: Initialized, Role: FOLLOWER I0507 06:31:18.787801 5794 consensus_queue.cc:229] T ea48f25f7b394ecf8f30f2edcc242b0c P a4e7f434e44448c1a89a6436e7c5da3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4e7f434e44448c1a89a6436e7c5da3a" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 34093 } } peers { permanent_uuid: "4df733e8c0ed4f95b435c24c1f59c84c" member_type: VOTER last_known_addr { host: "127.0.57.196" port: 40131 } } peers { permanent_uuid: "57b5634a23754ddfbf10759f1def9b8d" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 38629 } } I0507 06:31:18.788271 5792 consensus_queue.cc:229] T ea48f25f7b394ecf8f30f2edcc242b0c P 57b5634a23754ddfbf10759f1def9b8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4e7f434e44448c1a89a6436e7c5da3a" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 34093 } } peers { permanent_uuid: "4df733e8c0ed4f95b435c24c1f59c84c" member_type: VOTER last_known_addr { host: "127.0.57.196" port: 40131 } } peers { permanent_uuid: "57b5634a23754ddfbf10759f1def9b8d" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 38629 } } I0507 06:31:18.788740 5793 ts_tablet_manager.cc:1081] T ea48f25f7b394ecf8f30f2edcc242b0c P 4df733e8c0ed4f95b435c24c1f59c84c: Time spent starting tablet: real 0.009s user 0.009s sys 0.000s I0507 06:31:18.788821 5650 heartbeater.cc:467] Master 127.0.57.254:37907 was elected leader, sending a full tablet report... I0507 06:31:18.789398 5528 heartbeater.cc:467] Master 127.0.57.254:37907 was elected leader, sending a full tablet report... I0507 06:31:18.789543 5794 ts_tablet_manager.cc:1081] T ea48f25f7b394ecf8f30f2edcc242b0c P a4e7f434e44448c1a89a6436e7c5da3a: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s I0507 06:31:18.790422 5792 ts_tablet_manager.cc:1081] T ea48f25f7b394ecf8f30f2edcc242b0c P 57b5634a23754ddfbf10759f1def9b8d: Time spent starting tablet: real 0.009s user 0.006s sys 0.002s W0507 06:31:18.877677 5529 tablet.cc:1786] T ea48f25f7b394ecf8f30f2edcc242b0c P 57b5634a23754ddfbf10759f1def9b8d: Can't schedule compaction. Clean time has not been advanced past its initial value. I0507 06:31:18.898406 5405 heartbeater.cc:467] Master 127.0.57.254:37907 was elected leader, sending a full tablet report... W0507 06:31:18.916164 5651 tablet.cc:1786] T ea48f25f7b394ecf8f30f2edcc242b0c P a4e7f434e44448c1a89a6436e7c5da3a: Can't schedule compaction. Clean time has not been advanced past its initial value. W0507 06:31:18.944555 5773 tablet.cc:1786] T ea48f25f7b394ecf8f30f2edcc242b0c P 4df733e8c0ed4f95b435c24c1f59c84c: Can't schedule compaction. Clean time has not been advanced past its initial value. I0507 06:31:18.994390 5798 raft_consensus.cc:472] T ea48f25f7b394ecf8f30f2edcc242b0c P 4df733e8c0ed4f95b435c24c1f59c84c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0507 06:31:18.994630 5798 raft_consensus.cc:494] T ea48f25f7b394ecf8f30f2edcc242b0c P 4df733e8c0ed4f95b435c24c1f59c84c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4e7f434e44448c1a89a6436e7c5da3a" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 34093 } } peers { permanent_uuid: "4df733e8c0ed4f95b435c24c1f59c84c" member_type: VOTER last_known_addr { host: "127.0.57.196" port: 40131 } } peers { permanent_uuid: "57b5634a23754ddfbf10759f1def9b8d" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 38629 } } I0507 06:31:18.995977 5798 leader_election.cc:296] T ea48f25f7b394ecf8f30f2edcc242b0c P 4df733e8c0ed4f95b435c24c1f59c84c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a4e7f434e44448c1a89a6436e7c5da3a (127.0.57.195:34093), 57b5634a23754ddfbf10759f1def9b8d (127.0.57.194:38629) I0507 06:31:19.001781 5607 tablet_service.cc:1286] Received RequestConsensusVote() RPC: tablet_id: "ea48f25f7b394ecf8f30f2edcc242b0c" candidate_uuid: "4df733e8c0ed4f95b435c24c1f59c84c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4e7f434e44448c1a89a6436e7c5da3a" is_pre_election: true I0507 06:31:19.002072 5607 raft_consensus.cc:2346] T ea48f25f7b394ecf8f30f2edcc242b0c P a4e7f434e44448c1a89a6436e7c5da3a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4df733e8c0ed4f95b435c24c1f59c84c in term 0. I0507 06:31:19.002161 5485 tablet_service.cc:1286] Received RequestConsensusVote() RPC: tablet_id: "ea48f25f7b394ecf8f30f2edcc242b0c" candidate_uuid: "4df733e8c0ed4f95b435c24c1f59c84c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57b5634a23754ddfbf10759f1def9b8d" is_pre_election: true I0507 06:31:19.002513 5485 raft_consensus.cc:2346] T ea48f25f7b394ecf8f30f2edcc242b0c P 57b5634a23754ddfbf10759f1def9b8d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4df733e8c0ed4f95b435c24c1f59c84c in term 0. I0507 06:31:19.002719 5661 leader_election.cc:310] T ea48f25f7b394ecf8f30f2edcc242b0c P 4df733e8c0ed4f95b435c24c1f59c84c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4df733e8c0ed4f95b435c24c1f59c84c, a4e7f434e44448c1a89a6436e7c5da3a; no voters: I0507 06:31:19.003124 5798 raft_consensus.cc:2640] T ea48f25f7b394ecf8f30f2edcc242b0c P 4df733e8c0ed4f95b435c24c1f59c84c [term 0 FOLLOWER]: Leader pre-election won for term 1 I0507 06:31:19.003216 5798 raft_consensus.cc:472] T ea48f25f7b394ecf8f30f2edcc242b0c P 4df733e8c0ed4f95b435c24c1f59c84c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0507 06:31:19.003269 5798 raft_consensus.cc:2886] T ea48f25f7b394ecf8f30f2edcc242b0c P 4df733e8c0ed4f95b435c24c1f59c84c [term 0 FOLLOWER]: Advancing to term 1 I0507 06:31:19.005295 5798 raft_consensus.cc:494] T ea48f25f7b394ecf8f30f2edcc242b0c P 4df733e8c0ed4f95b435c24c1f59c84c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4e7f434e44448c1a89a6436e7c5da3a" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 34093 } } peers { permanent_uuid: "4df733e8c0ed4f95b435c24c1f59c84c" member_type: VOTER last_known_addr { host: "127.0.57.196" port: 40131 } } peers { permanent_uuid: "57b5634a23754ddfbf10759f1def9b8d" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 38629 } } I0507 06:31:19.005857 5798 leader_election.cc:296] T ea48f25f7b394ecf8f30f2edcc242b0c P 4df733e8c0ed4f95b435c24c1f59c84c [CANDIDATE]: Term 1 election: Requested vote from peers a4e7f434e44448c1a89a6436e7c5da3a (127.0.57.195:34093), 57b5634a23754ddfbf10759f1def9b8d (127.0.57.194:38629) I0507 06:31:19.006352 5607 tablet_service.cc:1286] Received RequestConsensusVote() RPC: tablet_id: "ea48f25f7b394ecf8f30f2edcc242b0c" candidate_uuid: "4df733e8c0ed4f95b435c24c1f59c84c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4e7f434e44448c1a89a6436e7c5da3a" I0507 06:31:19.006407 5485 tablet_service.cc:1286] Received RequestConsensusVote() RPC: tablet_id: "ea48f25f7b394ecf8f30f2edcc242b0c" candidate_uuid: "4df733e8c0ed4f95b435c24c1f59c84c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57b5634a23754ddfbf10759f1def9b8d" I0507 06:31:19.006510 5607 raft_consensus.cc:2886] T ea48f25f7b394ecf8f30f2edcc242b0c P a4e7f434e44448c1a89a6436e7c5da3a [term 0 FOLLOWER]: Advancing to term 1 I0507 06:31:19.006541 5485 raft_consensus.cc:2886] T ea48f25f7b394ecf8f30f2edcc242b0c P 57b5634a23754ddfbf10759f1def9b8d [term 0 FOLLOWER]: Advancing to term 1 I0507 06:31:19.008251 5607 raft_consensus.cc:2346] T ea48f25f7b394ecf8f30f2edcc242b0c P a4e7f434e44448c1a89a6436e7c5da3a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4df733e8c0ed4f95b435c24c1f59c84c in term 1. I0507 06:31:19.008440 5485 raft_consensus.cc:2346] T ea48f25f7b394ecf8f30f2edcc242b0c P 57b5634a23754ddfbf10759f1def9b8d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4df733e8c0ed4f95b435c24c1f59c84c in term 1. I0507 06:31:19.008956 5661 leader_election.cc:310] T ea48f25f7b394ecf8f30f2edcc242b0c P 4df733e8c0ed4f95b435c24c1f59c84c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4df733e8c0ed4f95b435c24c1f59c84c, a4e7f434e44448c1a89a6436e7c5da3a; no voters: I0507 06:31:19.009320 5798 raft_consensus.cc:2640] T ea48f25f7b394ecf8f30f2edcc242b0c P 4df733e8c0ed4f95b435c24c1f59c84c [term 1 FOLLOWER]: Leader election won for term 1 I0507 06:31:19.012544 5798 raft_consensus.cc:667] T ea48f25f7b394ecf8f30f2edcc242b0c P 4df733e8c0ed4f95b435c24c1f59c84c [term 1 LEADER]: Becoming Leader. State: Replica: 4df733e8c0ed4f95b435c24c1f59c84c, State: Running, Role: LEADER I0507 06:31:19.012984 5798 consensus_queue.cc:206] T ea48f25f7b394ecf8f30f2edcc242b0c P 4df733e8c0ed4f95b435c24c1f59c84c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4e7f434e44448c1a89a6436e7c5da3a" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 34093 } } peers { permanent_uuid: "4df733e8c0ed4f95b435c24c1f59c84c" member_type: VOTER last_known_addr { host: "127.0.57.196" port: 40131 } } peers { permanent_uuid: "57b5634a23754ddfbf10759f1def9b8d" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 38629 } } I0507 06:31:19.017027 5241 catalog_manager.cc:3988] T ea48f25f7b394ecf8f30f2edcc242b0c P 4df733e8c0ed4f95b435c24c1f59c84c reported cstate change: term changed from 0 to 1, leader changed from to 4df733e8c0ed4f95b435c24c1f59c84c (127.0.57.196). New cstate: current_term: 1 leader_uuid: "4df733e8c0ed4f95b435c24c1f59c84c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4e7f434e44448c1a89a6436e7c5da3a" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 34093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4df733e8c0ed4f95b435c24c1f59c84c" member_type: VOTER last_known_addr { host: "127.0.57.196" port: 40131 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "57b5634a23754ddfbf10759f1def9b8d" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 38629 } health_report { overall_health: UNKNOWN } } } I0507 06:31:19.045737 231 external_mini_cluster.cc:623] 4 TS(s) registered with all masters I0507 06:31:19.047363 231 ts_itest-base.cc:232] Waiting for 0 tablets on tserver 3f283c2f720b44328ae90dbb6ede4f2c to finish bootstrapping I0507 06:31:19.055438 231 ts_itest-base.cc:232] Waiting for 1 tablets on tserver 57b5634a23754ddfbf10759f1def9b8d to finish bootstrapping I0507 06:31:19.061841 231 ts_itest-base.cc:232] Waiting for 1 tablets on tserver a4e7f434e44448c1a89a6436e7c5da3a to finish bootstrapping I0507 06:31:19.067684 231 ts_itest-base.cc:232] Waiting for 1 tablets on tserver 4df733e8c0ed4f95b435c24c1f59c84c to finish bootstrapping I0507 06:31:19.073928 231 external_mini_cluster.cc:1172] Attempting to check leaks for /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver pid 5653 I0507 06:31:19.074677 5769 generic_service.cc:171] Checking for leaks (request via RPC) I0507 06:31:19.122601 231 external_mini_cluster.cc:1131] Killing /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver with pid 5653 I0507 06:31:20.606384 5806 raft_consensus.cc:472] T ea48f25f7b394ecf8f30f2edcc242b0c P 57b5634a23754ddfbf10759f1def9b8d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0507 06:31:20.606580 5806 raft_consensus.cc:494] T ea48f25f7b394ecf8f30f2edcc242b0c P 57b5634a23754ddfbf10759f1def9b8d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4e7f434e44448c1a89a6436e7c5da3a" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 34093 } } peers { permanent_uuid: "4df733e8c0ed4f95b435c24c1f59c84c" member_type: VOTER last_known_addr { host: "127.0.57.196" port: 40131 } } peers { permanent_uuid: "57b5634a23754ddfbf10759f1def9b8d" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 38629 } } I0507 06:31:20.607990 5806 leader_election.cc:296] T ea48f25f7b394ecf8f30f2edcc242b0c P 57b5634a23754ddfbf10759f1def9b8d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a4e7f434e44448c1a89a6436e7c5da3a (127.0.57.195:34093), 4df733e8c0ed4f95b435c24c1f59c84c (127.0.57.196:40131) W0507 06:31:20.609648 5418 leader_election.cc:341] T ea48f25f7b394ecf8f30f2edcc242b0c P 57b5634a23754ddfbf10759f1def9b8d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4df733e8c0ed4f95b435c24c1f59c84c (127.0.57.196:40131): Network error: Client connection negotiation failed: client connection to 127.0.57.196:40131: connect: Connection refused (error 111) I0507 06:31:20.613940 5607 tablet_service.cc:1286] Received RequestConsensusVote() RPC: tablet_id: "ea48f25f7b394ecf8f30f2edcc242b0c" candidate_uuid: "57b5634a23754ddfbf10759f1def9b8d" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4e7f434e44448c1a89a6436e7c5da3a" is_pre_election: true I0507 06:31:20.614162 5607 raft_consensus.cc:2346] T ea48f25f7b394ecf8f30f2edcc242b0c P a4e7f434e44448c1a89a6436e7c5da3a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57b5634a23754ddfbf10759f1def9b8d in term 1. I0507 06:31:20.614624 5417 leader_election.cc:310] T ea48f25f7b394ecf8f30f2edcc242b0c P 57b5634a23754ddfbf10759f1def9b8d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 57b5634a23754ddfbf10759f1def9b8d, a4e7f434e44448c1a89a6436e7c5da3a; no voters: 4df733e8c0ed4f95b435c24c1f59c84c I0507 06:31:20.615085 5806 raft_consensus.cc:2640] T ea48f25f7b394ecf8f30f2edcc242b0c P 57b5634a23754ddfbf10759f1def9b8d [term 1 FOLLOWER]: Leader pre-election won for term 2 I0507 06:31:20.615171 5806 raft_consensus.cc:472] T ea48f25f7b394ecf8f30f2edcc242b0c P 57b5634a23754ddfbf10759f1def9b8d [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0507 06:31:20.615263 5806 raft_consensus.cc:2886] T ea48f25f7b394ecf8f30f2edcc242b0c P 57b5634a23754ddfbf10759f1def9b8d [term 1 FOLLOWER]: Advancing to term 2 I0507 06:31:20.617194 5806 raft_consensus.cc:494] T ea48f25f7b394ecf8f30f2edcc242b0c P 57b5634a23754ddfbf10759f1def9b8d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4e7f434e44448c1a89a6436e7c5da3a" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 34093 } } peers { permanent_uuid: "4df733e8c0ed4f95b435c24c1f59c84c" member_type: VOTER last_known_addr { host: "127.0.57.196" port: 40131 } } peers { permanent_uuid: "57b5634a23754ddfbf10759f1def9b8d" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 38629 } } I0507 06:31:20.617765 5806 leader_election.cc:296] T ea48f25f7b394ecf8f30f2edcc242b0c P 57b5634a23754ddfbf10759f1def9b8d [CANDIDATE]: Term 2 election: Requested vote from peers a4e7f434e44448c1a89a6436e7c5da3a (127.0.57.195:34093), 4df733e8c0ed4f95b435c24c1f59c84c (127.0.57.196:40131) I0507 06:31:20.618243 5607 tablet_service.cc:1286] Received RequestConsensusVote() RPC: tablet_id: "ea48f25f7b394ecf8f30f2edcc242b0c" candidate_uuid: "57b5634a23754ddfbf10759f1def9b8d" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4e7f434e44448c1a89a6436e7c5da3a" W0507 06:31:20.618356 5418 leader_election.cc:341] T ea48f25f7b394ecf8f30f2edcc242b0c P 57b5634a23754ddfbf10759f1def9b8d [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 4df733e8c0ed4f95b435c24c1f59c84c (127.0.57.196:40131): Network error: Client connection negotiation failed: client connection to 127.0.57.196:40131: connect: Connection refused (error 111) I0507 06:31:20.618445 5607 raft_consensus.cc:2886] T ea48f25f7b394ecf8f30f2edcc242b0c P a4e7f434e44448c1a89a6436e7c5da3a [term 1 FOLLOWER]: Advancing to term 2 I0507 06:31:20.620213 5607 raft_consensus.cc:2346] T ea48f25f7b394ecf8f30f2edcc242b0c P a4e7f434e44448c1a89a6436e7c5da3a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57b5634a23754ddfbf10759f1def9b8d in term 2. I0507 06:31:20.620692 5417 leader_election.cc:310] T ea48f25f7b394ecf8f30f2edcc242b0c P 57b5634a23754ddfbf10759f1def9b8d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 57b5634a23754ddfbf10759f1def9b8d, a4e7f434e44448c1a89a6436e7c5da3a; no voters: 4df733e8c0ed4f95b435c24c1f59c84c I0507 06:31:20.620985 5806 raft_consensus.cc:2640] T ea48f25f7b394ecf8f30f2edcc242b0c P 57b5634a23754ddfbf10759f1def9b8d [term 2 FOLLOWER]: Leader election won for term 2 I0507 06:31:20.621349 5806 raft_consensus.cc:667] T ea48f25f7b394ecf8f30f2edcc242b0c P 57b5634a23754ddfbf10759f1def9b8d [term 2 LEADER]: Becoming Leader. State: Replica: 57b5634a23754ddfbf10759f1def9b8d, State: Running, Role: LEADER I0507 06:31:20.621785 5806 consensus_queue.cc:206] T ea48f25f7b394ecf8f30f2edcc242b0c P 57b5634a23754ddfbf10759f1def9b8d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4e7f434e44448c1a89a6436e7c5da3a" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 34093 } } peers { permanent_uuid: "4df733e8c0ed4f95b435c24c1f59c84c" member_type: VOTER last_known_addr { host: "127.0.57.196" port: 40131 } } peers { permanent_uuid: "57b5634a23754ddfbf10759f1def9b8d" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 38629 } } I0507 06:31:20.626714 5239 catalog_manager.cc:3988] T ea48f25f7b394ecf8f30f2edcc242b0c P 57b5634a23754ddfbf10759f1def9b8d reported cstate change: term changed from 1 to 2, leader changed from 4df733e8c0ed4f95b435c24c1f59c84c (127.0.57.196) to 57b5634a23754ddfbf10759f1def9b8d (127.0.57.194). New cstate: current_term: 2 leader_uuid: "57b5634a23754ddfbf10759f1def9b8d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4e7f434e44448c1a89a6436e7c5da3a" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 34093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4df733e8c0ed4f95b435c24c1f59c84c" member_type: VOTER last_known_addr { host: "127.0.57.196" port: 40131 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "57b5634a23754ddfbf10759f1def9b8d" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 38629 } health_report { overall_health: HEALTHY } } } I0507 06:31:21.182639 5607 raft_consensus.cc:1181] T ea48f25f7b394ecf8f30f2edcc242b0c P a4e7f434e44448c1a89a6436e7c5da3a [term 2 FOLLOWER]: Refusing update from remote peer 57b5634a23754ddfbf10759f1def9b8d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch) I0507 06:31:21.183624 5816 consensus_queue.cc:984] T ea48f25f7b394ecf8f30f2edcc242b0c P 57b5634a23754ddfbf10759f1def9b8d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4e7f434e44448c1a89a6436e7c5da3a" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 34093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s W0507 06:31:21.190398 5418 consensus_peers.cc:457] T ea48f25f7b394ecf8f30f2edcc242b0c P 57b5634a23754ddfbf10759f1def9b8d -> Peer 4df733e8c0ed4f95b435c24c1f59c84c (127.0.57.196:40131): Couldn't send request to peer 4df733e8c0ed4f95b435c24c1f59c84c. Status: Network error: Client connection negotiation failed: client connection to 127.0.57.196:40131: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry. W0507 06:31:23.590956 5418 consensus_peers.cc:457] T ea48f25f7b394ecf8f30f2edcc242b0c P 57b5634a23754ddfbf10759f1def9b8d -> Peer 4df733e8c0ed4f95b435c24c1f59c84c (127.0.57.196:40131): Couldn't send request to peer 4df733e8c0ed4f95b435c24c1f59c84c. Status: Network error: Client connection negotiation failed: client connection to 127.0.57.196:40131: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry. W0507 06:31:25.908819 5418 consensus_peers.cc:457] T ea48f25f7b394ecf8f30f2edcc242b0c P 57b5634a23754ddfbf10759f1def9b8d -> Peer 4df733e8c0ed4f95b435c24c1f59c84c (127.0.57.196:40131): Couldn't send request to peer 4df733e8c0ed4f95b435c24c1f59c84c. Status: Network error: Client connection negotiation failed: client connection to 127.0.57.196:40131: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry. W0507 06:31:28.451936 5418 consensus_peers.cc:457] T ea48f25f7b394ecf8f30f2edcc242b0c P 57b5634a23754ddfbf10759f1def9b8d -> Peer 4df733e8c0ed4f95b435c24c1f59c84c (127.0.57.196:40131): Couldn't send request to peer 4df733e8c0ed4f95b435c24c1f59c84c. Status: Network error: Client connection negotiation failed: client connection to 127.0.57.196:40131: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry. I0507 06:31:31.023051 5822 consensus_queue.cc:544] T ea48f25f7b394ecf8f30f2edcc242b0c P 57b5634a23754ddfbf10759f1def9b8d [LEADER]: Leader has been unable to successfully communicate with peer 4df733e8c0ed4f95b435c24c1f59c84c for more than 10 seconds (10.400s) W0507 06:31:31.024833 5418 consensus_peers.cc:457] T ea48f25f7b394ecf8f30f2edcc242b0c P 57b5634a23754ddfbf10759f1def9b8d -> Peer 4df733e8c0ed4f95b435c24c1f59c84c (127.0.57.196:40131): Couldn't send request to peer 4df733e8c0ed4f95b435c24c1f59c84c. Status: Network error: Client connection negotiation failed: client connection to 127.0.57.196:40131: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry. I0507 06:31:31.025902 5241 catalog_manager.cc:3616] Sending ChangeConfig:ADD_PEER:NON_VOTER on tablet ea48f25f7b394ecf8f30f2edcc242b0c (attempt 1) I0507 06:31:31.028317 5485 consensus_queue.cc:206] T ea48f25f7b394ecf8f30f2edcc242b0c P 57b5634a23754ddfbf10759f1def9b8d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a4e7f434e44448c1a89a6436e7c5da3a" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 34093 } } peers { permanent_uuid: "4df733e8c0ed4f95b435c24c1f59c84c" member_type: VOTER last_known_addr { host: "127.0.57.196" port: 40131 } } peers { permanent_uuid: "57b5634a23754ddfbf10759f1def9b8d" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 38629 } } peers { permanent_uuid: "3f283c2f720b44328ae90dbb6ede4f2c" member_type: NON_VOTER last_known_addr { host: "127.0.57.193" port: 39837 } attrs { promote: true } } I0507 06:31:31.031342 5607 raft_consensus.cc:1181] T ea48f25f7b394ecf8f30f2edcc242b0c P a4e7f434e44448c1a89a6436e7c5da3a [term 2 FOLLOWER]: Refusing update from remote peer 57b5634a23754ddfbf10759f1def9b8d: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch) I0507 06:31:31.032291 5837 consensus_queue.cc:984] T ea48f25f7b394ecf8f30f2edcc242b0c P 57b5634a23754ddfbf10759f1def9b8d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4e7f434e44448c1a89a6436e7c5da3a" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 34093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s W0507 06:31:31.032634 5418 consensus_peers.cc:457] T ea48f25f7b394ecf8f30f2edcc242b0c P 57b5634a23754ddfbf10759f1def9b8d -> Peer 4df733e8c0ed4f95b435c24c1f59c84c (127.0.57.196:40131): Couldn't send request to peer 4df733e8c0ed4f95b435c24c1f59c84c. Status: Network error: Client connection negotiation failed: client connection to 127.0.57.196:40131: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry. W0507 06:31:31.038622 5417 consensus_peers.cc:457] T ea48f25f7b394ecf8f30f2edcc242b0c P 57b5634a23754ddfbf10759f1def9b8d -> Peer 3f283c2f720b44328ae90dbb6ede4f2c (127.0.57.193:39837): Couldn't send request to peer 3f283c2f720b44328ae90dbb6ede4f2c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ea48f25f7b394ecf8f30f2edcc242b0c. This is attempt 1: this message will repeat every 5th retry. I0507 06:31:31.040802 5834 raft_consensus.cc:2788] T ea48f25f7b394ecf8f30f2edcc242b0c P 57b5634a23754ddfbf10759f1def9b8d [term 2 LEADER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER 3f283c2f720b44328ae90dbb6ede4f2c (127.0.57.193) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a4e7f434e44448c1a89a6436e7c5da3a" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 34093 } } peers { permanent_uuid: "4df733e8c0ed4f95b435c24c1f59c84c" member_type: VOTER last_known_addr { host: "127.0.57.196" port: 40131 } } peers { permanent_uuid: "57b5634a23754ddfbf10759f1def9b8d" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 38629 } } peers { permanent_uuid: "3f283c2f720b44328ae90dbb6ede4f2c" member_type: NON_VOTER last_known_addr { host: "127.0.57.193" port: 39837 } attrs { promote: true } } } I0507 06:31:31.041460 5607 raft_consensus.cc:2788] T ea48f25f7b394ecf8f30f2edcc242b0c P a4e7f434e44448c1a89a6436e7c5da3a [term 2 FOLLOWER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER 3f283c2f720b44328ae90dbb6ede4f2c (127.0.57.193) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a4e7f434e44448c1a89a6436e7c5da3a" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 34093 } } peers { permanent_uuid: "4df733e8c0ed4f95b435c24c1f59c84c" member_type: VOTER last_known_addr { host: "127.0.57.196" port: 40131 } } peers { permanent_uuid: "57b5634a23754ddfbf10759f1def9b8d" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 38629 } } peers { permanent_uuid: "3f283c2f720b44328ae90dbb6ede4f2c" member_type: NON_VOTER last_known_addr { host: "127.0.57.193" port: 39837 } attrs { promote: true } } } I0507 06:31:31.044811 5222 catalog_manager.cc:3531] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet ea48f25f7b394ecf8f30f2edcc242b0c with cas_config_opid_index -1: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1) I0507 06:31:31.046177 5241 catalog_manager.cc:3988] T ea48f25f7b394ecf8f30f2edcc242b0c P 57b5634a23754ddfbf10759f1def9b8d reported cstate change: config changed from index -1 to 2, NON_VOTER 3f283c2f720b44328ae90dbb6ede4f2c (127.0.57.193) added. New cstate: current_term: 2 leader_uuid: "57b5634a23754ddfbf10759f1def9b8d" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a4e7f434e44448c1a89a6436e7c5da3a" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 34093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4df733e8c0ed4f95b435c24c1f59c84c" member_type: VOTER last_known_addr { host: "127.0.57.196" port: 40131 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "57b5634a23754ddfbf10759f1def9b8d" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 38629 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3f283c2f720b44328ae90dbb6ede4f2c" member_type: NON_VOTER last_known_addr { host: "127.0.57.193" port: 39837 } attrs { promote: true } health_report { overall_health: UNKNOWN } } } I0507 06:31:31.060904 231 external_mini_cluster.cc:1172] Attempting to check leaks for /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver pid 5275 I0507 06:31:31.062788 5402 generic_service.cc:171] Checking for leaks (request via RPC) I0507 06:31:31.110649 231 external_mini_cluster.cc:1131] Killing /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver with pid 5275 I0507 06:31:31.122638 231 external_mini_cluster.cc:1172] Attempting to check leaks for /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver pid 5409 I0507 06:31:31.123808 5525 generic_service.cc:171] Checking for leaks (request via RPC) I0507 06:31:31.182938 231 external_mini_cluster.cc:1131] Killing /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver with pid 5409 I0507 06:31:31.193943 231 external_mini_cluster.cc:1172] Attempting to check leaks for /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver pid 5531 I0507 06:31:31.195132 5647 generic_service.cc:171] Checking for leaks (request via RPC) I0507 06:31:31.252427 231 external_mini_cluster.cc:1131] Killing /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver with pid 5531 I0507 06:31:31.262796 231 external_mini_cluster.cc:1172] Attempting to check leaks for /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-master pid 5215 I0507 06:31:31.264014 5271 generic_service.cc:171] Checking for leaks (request via RPC) I0507 06:31:31.354326 231 external_mini_cluster.cc:1131] Killing /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-master with pid 5215 [ OK ] RaftConsensusNonVoterITest.CatalogManagerAddsNonVoter (14158 ms) [ RUN ] RaftConsensusNonVoterITest.TabletServerIsGoneAndBack I0507 06:31:31.378026 231 test_util.cc:212] Using random seed: 1578736455 I0507 06:31:31.379633 231 ts_itest-base.cc:112] Starting cluster with: I0507 06:31:31.379684 231 ts_itest-base.cc:113] -------------- I0507 06:31:31.379703 231 ts_itest-base.cc:114] 4 tablet servers I0507 06:31:31.379739 231 ts_itest-base.cc:115] 3 replicas per TS I0507 06:31:31.379757 231 ts_itest-base.cc:116] -------------- I0507 06:31:31.381165 231 external_mini_cluster.cc:933] Running /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-master /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-master --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.TabletServerIsGoneAndBack.1557210500371863-231/raft_consensus-itest-cluster/master-0/wal --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.TabletServerIsGoneAndBack.1557210500371863-231/raft_consensus-itest-cluster/master-0/data --block_manager=log --webserver_interface=localhost --ipki_ca_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.0.57.254:36133 --webserver_interface=127.0.57.254 --webserver_port=0 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.TabletServerIsGoneAndBack.1557210500371863-231/raft_consensus-itest-cluster/master-0/logs --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.TabletServerIsGoneAndBack.1557210500371863-231/raft_consensus-itest-cluster/master-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=true --rpc_reuseport=true with env {} WARNING: Logging before InitGoogleLogging() is written to STDERR W0507 06:31:31.591945 5855 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0507 06:31:31.592190 5855 flags.cc:404] Enabled unsafe flag: --never_fsync=true W0507 06:31:31.594792 5855 flags.cc:404] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true W0507 06:31:31.594856 5855 flags.cc:404] Enabled experimental flag: --ipki_ca_key_size=1024 W0507 06:31:31.594897 5855 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W0507 06:31:31.594923 5855 flags.cc:404] Enabled experimental flag: --tsk_num_rsa_bits=512 W0507 06:31:31.594964 5855 flags.cc:404] Enabled experimental flag: --rpc_reuseport=true I0507 06:31:31.599792 5855 master_main.cc:73] Master server non-default flags: --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.TabletServerIsGoneAndBack.1557210500371863-231/raft_consensus-itest-cluster/master-0/data --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.TabletServerIsGoneAndBack.1557210500371863-231/raft_consensus-itest-cluster/master-0/wal --ipki_ca_key_size=1024 --ipki_server_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.0.57.254:36133 --rpc_reuseport=true --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.TabletServerIsGoneAndBack.1557210500371863-231/raft_consensus-itest-cluster/master-0/data/info.pb --webserver_interface=127.0.57.254 --webserver_port=0 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.TabletServerIsGoneAndBack.1557210500371863-231/raft_consensus-itest-cluster/master-0/logs --logbuflevel=-1 --logtostderr=true Master server version: kudu 1.10.0-cdh6.x-SNAPSHOT revision 7d0ee26697926e87af9ef395008799577f6ab43f build type FASTDEBUG built by jenkins at 06 May 2019 22:44:21 PST on quasar-knmpfx-1.vpc.cloudera.com build id 322017 ASAN enabled I0507 06:31:31.607075 5855 system_ntp.cc:202] NTP initialized. Skew: 500ppm Current error: 120016us I0507 06:31:31.608438 5855 fs_manager.cc:263] Metadata directory not provided I0507 06:31:31.608515 5855 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0507 06:31:31.608866 5855 server_base.cc:433] Could not load existing FS layout: Not found: could not find a healthy instance file I0507 06:31:31.608942 5855 server_base.cc:434] Attempting to create new FS layout instead I0507 06:31:31.611605 5855 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.TabletServerIsGoneAndBack.1557210500371863-231/raft_consensus-itest-cluster/master-0/data/instance: uuid: "de7e5da07e2344d7a0645b121021b617" format_stamp: "Formatted at 2019-05-07 06:31:31 on dist-test-slave-dist-test-slave-rqp6" I0507 06:31:31.612148 5855 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.TabletServerIsGoneAndBack.1557210500371863-231/raft_consensus-itest-cluster/master-0/wal/instance: uuid: "de7e5da07e2344d7a0645b121021b617" format_stamp: "Formatted at 2019-05-07 06:31:31 on dist-test-slave-dist-test-slave-rqp6" I0507 06:31:31.619968 5855 fs_manager.cc:503] Time spent creating directory manager: real 0.008s user 0.009s sys 0.001s I0507 06:31:31.621410 5855 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0507 06:31:31.622141 5855 file_cache.cc:466] Constructed file cache lbm with capacity 419430 I0507 06:31:31.625823 5855 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s I0507 06:31:31.625907 5855 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.TabletServerIsGoneAndBack.1557210500371863-231/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.TabletServerIsGoneAndBack.1557210500371863-231/raft_consensus-itest-cluster/master-0/wal uuid: "de7e5da07e2344d7a0645b121021b617" format_stamp: "Formatted at 2019-05-07 06:31:31 on dist-test-slave-dist-test-slave-rqp6" I0507 06:31:31.626047 5855 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.TabletServerIsGoneAndBack.1557210500371863-231/raft_consensus-itest-cluster/master-0/wal metadata directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.TabletServerIsGoneAndBack.1557210500371863-231/raft_consensus-itest-cluster/master-0/wal 1 data directories: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.TabletServerIsGoneAndBack.1557210500371863-231/raft_consensus-itest-cluster/master-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0507 06:31:31.660913 5855 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0507 06:31:31.675969 5855 rpc_server.cc:205] RPC server started. Bound to: 127.0.57.254:36133 I0507 06:31:31.677242 5855 webserver.cc:308] Webserver started at http://127.0.57.254:44167/ using document root and password file I0507 06:31:31.678130 5855 server_base.cc:608] Dumped server information to /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.TabletServerIsGoneAndBack.1557210500371863-231/raft_consensus-itest-cluster/master-0/data/info.pb I0507 06:31:31.682343 5914 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0507 06:31:31.688385 231 external_mini_cluster.cc:995] Started /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-master as pid 5855 I0507 06:31:31.689250 231 external_mini_cluster.cc:933] Running /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.TabletServerIsGoneAndBack.1557210500371863-231/raft_consensus-itest-cluster/ts-0/wal --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.TabletServerIsGoneAndBack.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data --block_manager=log --rpc_bind_addresses=127.0.57.193:0 --local_ip_for_outbound_sockets=127.0.57.193 --webserver_interface=127.0.57.193 --webserver_port=0 --tserver_master_addrs=127.0.57.254:36133 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.TabletServerIsGoneAndBack.1557210500371863-231/raft_consensus-itest-cluster/ts-0/logs --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.TabletServerIsGoneAndBack.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=true --follower_unavailable_considered_failed_sec=5 --tablet_copy_download_file_inject_latency_ms=180000 --consensus_rpc_timeout_ms=5000 with env {} I0507 06:31:31.693367 5914 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P de7e5da07e2344d7a0645b121021b617: Bootstrap starting. I0507 06:31:31.696066 5914 tablet_bootstrap.cc:590] T 00000000000000000000000000000000 P de7e5da07e2344d7a0645b121021b617: No blocks or log segments found. Creating new log. I0507 06:31:31.701313 5914 log.cc:525] T 00000000000000000000000000000000 P de7e5da07e2344d7a0645b121021b617: Log is configured to *not* fsync() on all Append() calls I0507 06:31:31.706354 5914 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P de7e5da07e2344d7a0645b121021b617: No bootstrap required, opened a new log I0507 06:31:31.714103 5914 raft_consensus.cc:340] T 00000000000000000000000000000000 P de7e5da07e2344d7a0645b121021b617 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de7e5da07e2344d7a0645b121021b617" member_type: VOTER } I0507 06:31:31.714385 5914 raft_consensus.cc:366] T 00000000000000000000000000000000 P de7e5da07e2344d7a0645b121021b617 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0507 06:31:31.714473 5914 raft_consensus.cc:705] T 00000000000000000000000000000000 P de7e5da07e2344d7a0645b121021b617 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de7e5da07e2344d7a0645b121021b617, State: Initialized, Role: FOLLOWER I0507 06:31:31.715085 5914 consensus_queue.cc:229] T 00000000000000000000000000000000 P de7e5da07e2344d7a0645b121021b617 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de7e5da07e2344d7a0645b121021b617" member_type: VOTER } I0507 06:31:31.715272 5914 raft_consensus.cc:380] T 00000000000000000000000000000000 P de7e5da07e2344d7a0645b121021b617 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0507 06:31:31.715344 5914 raft_consensus.cc:472] T 00000000000000000000000000000000 P de7e5da07e2344d7a0645b121021b617 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0507 06:31:31.715440 5914 raft_consensus.cc:2886] T 00000000000000000000000000000000 P de7e5da07e2344d7a0645b121021b617 [term 0 FOLLOWER]: Advancing to term 1 I0507 06:31:31.717273 5914 raft_consensus.cc:494] T 00000000000000000000000000000000 P de7e5da07e2344d7a0645b121021b617 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de7e5da07e2344d7a0645b121021b617" member_type: VOTER } I0507 06:31:31.717768 5914 leader_election.cc:310] T 00000000000000000000000000000000 P de7e5da07e2344d7a0645b121021b617 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: de7e5da07e2344d7a0645b121021b617; no voters: I0507 06:31:31.718677 5917 raft_consensus.cc:2640] T 00000000000000000000000000000000 P de7e5da07e2344d7a0645b121021b617 [term 1 FOLLOWER]: Leader election won for term 1 I0507 06:31:31.719038 5914 leader_election.cc:296] T 00000000000000000000000000000000 P de7e5da07e2344d7a0645b121021b617 [CANDIDATE]: Term 1 election: Requested vote from peers I0507 06:31:31.719914 5917 raft_consensus.cc:667] T 00000000000000000000000000000000 P de7e5da07e2344d7a0645b121021b617 [term 1 LEADER]: Becoming Leader. State: Replica: de7e5da07e2344d7a0645b121021b617, State: Running, Role: LEADER I0507 06:31:31.720340 5917 consensus_queue.cc:206] T 00000000000000000000000000000000 P de7e5da07e2344d7a0645b121021b617 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de7e5da07e2344d7a0645b121021b617" member_type: VOTER } I0507 06:31:31.724081 5918 sys_catalog.cc:337] T 00000000000000000000000000000000 P de7e5da07e2344d7a0645b121021b617 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "de7e5da07e2344d7a0645b121021b617" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de7e5da07e2344d7a0645b121021b617" member_type: VOTER } } I0507 06:31:31.724612 5918 sys_catalog.cc:340] T 00000000000000000000000000000000 P de7e5da07e2344d7a0645b121021b617 [sys.catalog]: This master's current role is: LEADER I0507 06:31:31.724629 5914 sys_catalog.cc:337] T 00000000000000000000000000000000 P de7e5da07e2344d7a0645b121021b617 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 1 leader_uuid: "de7e5da07e2344d7a0645b121021b617" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de7e5da07e2344d7a0645b121021b617" member_type: VOTER } } I0507 06:31:31.725006 5914 sys_catalog.cc:340] T 00000000000000000000000000000000 P de7e5da07e2344d7a0645b121021b617 [sys.catalog]: This master's current role is: LEADER I0507 06:31:31.724607 5919 sys_catalog.cc:337] T 00000000000000000000000000000000 P de7e5da07e2344d7a0645b121021b617 [sys.catalog]: SysCatalogTable state changed. Reason: New leader de7e5da07e2344d7a0645b121021b617. Latest consensus state: current_term: 1 leader_uuid: "de7e5da07e2344d7a0645b121021b617" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de7e5da07e2344d7a0645b121021b617" member_type: VOTER } } I0507 06:31:31.725409 5919 sys_catalog.cc:340] T 00000000000000000000000000000000 P de7e5da07e2344d7a0645b121021b617 [sys.catalog]: This master's current role is: LEADER I0507 06:31:31.726681 5914 sys_catalog.cc:424] T 00000000000000000000000000000000 P de7e5da07e2344d7a0645b121021b617 [sys.catalog]: configured and running, proceeding with master startup. W0507 06:31:31.734277 5924 catalog_manager.cc:1113] T 00000000000000000000000000000000 P de7e5da07e2344d7a0645b121021b617: acquiring CA information for follower catalog manager: Not found: root CA entry not found W0507 06:31:31.734364 5924 catalog_manager.cc:596] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry I0507 06:31:31.734467 5923 catalog_manager.cc:1055] Loading table and tablet metadata into memory... I0507 06:31:31.736281 5923 catalog_manager.cc:1066] Initializing Kudu internal certificate authority... I0507 06:31:31.766489 5923 catalog_manager.cc:950] Generated new certificate authority record I0507 06:31:31.767855 5923 catalog_manager.cc:1075] Loading token signing keys... I0507 06:31:31.777907 5923 catalog_manager.cc:4292] T 00000000000000000000000000000000 P de7e5da07e2344d7a0645b121021b617: Generated new TSK 0 WARNING: Logging before InitGoogleLogging() is written to STDERR W0507 06:31:31.855445 5915 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0507 06:31:31.855687 5915 flags.cc:404] Enabled unsafe flag: --never_fsync=true W0507 06:31:31.857841 5915 flags.cc:404] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true W0507 06:31:31.857906 5915 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W0507 06:31:31.858005 5915 flags.cc:404] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.57.193 I0507 06:31:31.862630 5915 tablet_server_main.cc:78] Tablet server non-default flags: --consensus_rpc_timeout_ms=5000 --follower_unavailable_considered_failed_sec=5 --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.TabletServerIsGoneAndBack.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.TabletServerIsGoneAndBack.1557210500371863-231/raft_consensus-itest-cluster/ts-0/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.0.57.193:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.TabletServerIsGoneAndBack.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data/info.pb --webserver_interface=127.0.57.193 --webserver_port=0 --tablet_copy_download_file_inject_latency_ms=180000 --tserver_master_addrs=127.0.57.254:36133 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.0.57.193 --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.TabletServerIsGoneAndBack.1557210500371863-231/raft_consensus-itest-cluster/ts-0/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.10.0-cdh6.x-SNAPSHOT revision 7d0ee26697926e87af9ef395008799577f6ab43f build type FASTDEBUG built by jenkins at 06 May 2019 22:44:21 PST on quasar-knmpfx-1.vpc.cloudera.com build id 322017 ASAN enabled I0507 06:31:31.870086 5915 system_ntp.cc:202] NTP initialized. Skew: 500ppm Current error: 120016us I0507 06:31:31.870751 5915 fs_manager.cc:263] Metadata directory not provided I0507 06:31:31.870817 5915 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0507 06:31:31.871068 5915 server_base.cc:433] Could not load existing FS layout: Not found: could not find a healthy instance file I0507 06:31:31.871136 5915 server_base.cc:434] Attempting to create new FS layout instead I0507 06:31:31.873492 5915 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.TabletServerIsGoneAndBack.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data/instance: uuid: "bc60338af9984198a753a6394960edcd" format_stamp: "Formatted at 2019-05-07 06:31:31 on dist-test-slave-dist-test-slave-rqp6" I0507 06:31:31.874034 5915 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.TabletServerIsGoneAndBack.1557210500371863-231/raft_consensus-itest-cluster/ts-0/wal/instance: uuid: "bc60338af9984198a753a6394960edcd" format_stamp: "Formatted at 2019-05-07 06:31:31 on dist-test-slave-dist-test-slave-rqp6" I0507 06:31:31.883244 5915 fs_manager.cc:503] Time spent creating directory manager: real 0.009s user 0.008s sys 0.001s I0507 06:31:31.884349 5915 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0507 06:31:31.884972 5915 file_cache.cc:466] Constructed file cache lbm with capacity 419430 I0507 06:31:31.888908 5915 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s I0507 06:31:31.889003 5915 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.TabletServerIsGoneAndBack.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.TabletServerIsGoneAndBack.1557210500371863-231/raft_consensus-itest-cluster/ts-0/wal uuid: "bc60338af9984198a753a6394960edcd" format_stamp: "Formatted at 2019-05-07 06:31:31 on dist-test-slave-dist-test-slave-rqp6" I0507 06:31:31.889158 5915 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.TabletServerIsGoneAndBack.1557210500371863-231/raft_consensus-itest-cluster/ts-0/wal metadata directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.TabletServerIsGoneAndBack.1557210500371863-231/raft_consensus-itest-cluster/ts-0/wal 1 data directories: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.TabletServerIsGoneAndBack.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0507 06:31:31.921752 5915 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0507 06:31:31.923414 5915 ts_tablet_manager.cc:356] Loaded tablet metadata (0 total tablets, 0 live tablets) I0507 06:31:31.923503 5915 ts_tablet_manager.cc:376] Registered 0 tablets I0507 06:31:32.042763 5915 rpc_server.cc:205] RPC server started. Bound to: 127.0.57.193:33675 I0507 06:31:32.043933 5915 webserver.cc:308] Webserver started at http://127.0.57.193:45487/ using document root and password file I0507 06:31:32.044616 5915 server_base.cc:608] Dumped server information to /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.TabletServerIsGoneAndBack.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data/info.pb I0507 06:31:32.046422 231 external_mini_cluster.cc:995] Started /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver as pid 5915 I0507 06:31:32.047310 231 external_mini_cluster.cc:933] Running /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.TabletServerIsGoneAndBack.1557210500371863-231/raft_consensus-itest-cluster/ts-1/wal --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.TabletServerIsGoneAndBack.1557210500371863-231/raft_consensus-itest-cluster/ts-1/data --block_manager=log --rpc_bind_addresses=127.0.57.194:0 --local_ip_for_outbound_sockets=127.0.57.194 --webserver_interface=127.0.57.194 --webserver_port=0 --tserver_master_addrs=127.0.57.254:36133 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.TabletServerIsGoneAndBack.1557210500371863-231/raft_consensus-itest-cluster/ts-1/logs --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.TabletServerIsGoneAndBack.1557210500371863-231/raft_consensus-itest-cluster/ts-1/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=true --follower_unavailable_considered_failed_sec=5 --tablet_copy_download_file_inject_latency_ms=180000 --consensus_rpc_timeout_ms=5000 with env {} I0507 06:31:32.071494 6045 heartbeater.cc:345] Connected to a master server at 127.0.57.254:36133 I0507 06:31:32.071730 6045 heartbeater.cc:426] Registering TS with master... I0507 06:31:32.072332 6045 heartbeater.cc:475] Master 127.0.57.254:36133 requested a full tablet report, sending... I0507 06:31:32.074301 5881 ts_manager.cc:159] Registered new tserver with Master: bc60338af9984198a753a6394960edcd (127.0.57.193:33675) I0507 06:31:32.075960 5881 master_service.cc:273] Signed X509 certificate for tserver {username='slave'} at 127.0.57.193:37403 WARNING: Logging before InitGoogleLogging() is written to STDERR W0507 06:31:32.219033 6049 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0507 06:31:32.219277 6049 flags.cc:404] Enabled unsafe flag: --never_fsync=true W0507 06:31:32.221559 6049 flags.cc:404] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true W0507 06:31:32.221621 6049 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W0507 06:31:32.221722 6049 flags.cc:404] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.57.194 I0507 06:31:32.226163 6049 tablet_server_main.cc:78] Tablet server non-default flags: --consensus_rpc_timeout_ms=5000 --follower_unavailable_considered_failed_sec=5 --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.TabletServerIsGoneAndBack.1557210500371863-231/raft_consensus-itest-cluster/ts-1/data --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.TabletServerIsGoneAndBack.1557210500371863-231/raft_consensus-itest-cluster/ts-1/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.0.57.194:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.TabletServerIsGoneAndBack.1557210500371863-231/raft_consensus-itest-cluster/ts-1/data/info.pb --webserver_interface=127.0.57.194 --webserver_port=0 --tablet_copy_download_file_inject_latency_ms=180000 --tserver_master_addrs=127.0.57.254:36133 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.0.57.194 --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.TabletServerIsGoneAndBack.1557210500371863-231/raft_consensus-itest-cluster/ts-1/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.10.0-cdh6.x-SNAPSHOT revision 7d0ee26697926e87af9ef395008799577f6ab43f build type FASTDEBUG built by jenkins at 06 May 2019 22:44:21 PST on quasar-knmpfx-1.vpc.cloudera.com build id 322017 ASAN enabled I0507 06:31:32.234673 6049 system_ntp.cc:202] NTP initialized. Skew: 500ppm Current error: 120516us I0507 06:31:32.235476 6049 fs_manager.cc:263] Metadata directory not provided I0507 06:31:32.235535 6049 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0507 06:31:32.235886 6049 server_base.cc:433] Could not load existing FS layout: Not found: could not find a healthy instance file I0507 06:31:32.235968 6049 server_base.cc:434] Attempting to create new FS layout instead I0507 06:31:32.238626 6049 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.TabletServerIsGoneAndBack.1557210500371863-231/raft_consensus-itest-cluster/ts-1/data/instance: uuid: "f82b67eec9cc44e0811eadb4fb49e7b3" format_stamp: "Formatted at 2019-05-07 06:31:32 on dist-test-slave-dist-test-slave-rqp6" I0507 06:31:32.239219 6049 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.TabletServerIsGoneAndBack.1557210500371863-231/raft_consensus-itest-cluster/ts-1/wal/instance: uuid: "f82b67eec9cc44e0811eadb4fb49e7b3" format_stamp: "Formatted at 2019-05-07 06:31:32 on dist-test-slave-dist-test-slave-rqp6" I0507 06:31:32.247004 6049 fs_manager.cc:503] Time spent creating directory manager: real 0.008s user 0.006s sys 0.002s I0507 06:31:32.248108 6049 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0507 06:31:32.248818 6049 file_cache.cc:466] Constructed file cache lbm with capacity 419430 I0507 06:31:32.252068 6049 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s I0507 06:31:32.252157 6049 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.TabletServerIsGoneAndBack.1557210500371863-231/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.TabletServerIsGoneAndBack.1557210500371863-231/raft_consensus-itest-cluster/ts-1/wal uuid: "f82b67eec9cc44e0811eadb4fb49e7b3" format_stamp: "Formatted at 2019-05-07 06:31:32 on dist-test-slave-dist-test-slave-rqp6" I0507 06:31:32.252331 6049 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.TabletServerIsGoneAndBack.1557210500371863-231/raft_consensus-itest-cluster/ts-1/wal metadata directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.TabletServerIsGoneAndBack.1557210500371863-231/raft_consensus-itest-cluster/ts-1/wal 1 data directories: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.TabletServerIsGoneAndBack.1557210500371863-231/raft_consensus-itest-cluster/ts-1/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0507 06:31:32.298519 6049 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0507 06:31:32.300315 6049 ts_tablet_manager.cc:356] Loaded tablet metadata (0 total tablets, 0 live tablets) I0507 06:31:32.300392 6049 ts_tablet_manager.cc:376] Registered 0 tablets I0507 06:31:32.328193 6049 rpc_server.cc:205] RPC server started. Bound to: 127.0.57.194:42507 I0507 06:31:32.329515 6049 webserver.cc:308] Webserver started at http://127.0.57.194:39745/ using document root and password file I0507 06:31:32.330248 6049 server_base.cc:608] Dumped server information to /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.TabletServerIsGoneAndBack.1557210500371863-231/raft_consensus-itest-cluster/ts-1/data/info.pb I0507 06:31:32.338390 231 external_mini_cluster.cc:995] Started /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver as pid 6049 I0507 06:31:32.339179 231 external_mini_cluster.cc:933] Running /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.TabletServerIsGoneAndBack.1557210500371863-231/raft_consensus-itest-cluster/ts-2/wal --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.TabletServerIsGoneAndBack.1557210500371863-231/raft_consensus-itest-cluster/ts-2/data --block_manager=log --rpc_bind_addresses=127.0.57.195:0 --local_ip_for_outbound_sockets=127.0.57.195 --webserver_interface=127.0.57.195 --webserver_port=0 --tserver_master_addrs=127.0.57.254:36133 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.TabletServerIsGoneAndBack.1557210500371863-231/raft_consensus-itest-cluster/ts-2/logs --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.TabletServerIsGoneAndBack.1557210500371863-231/raft_consensus-itest-cluster/ts-2/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=true --follower_unavailable_considered_failed_sec=5 --tablet_copy_download_file_inject_latency_ms=180000 --consensus_rpc_timeout_ms=5000 with env {} I0507 06:31:32.344101 6168 heartbeater.cc:345] Connected to a master server at 127.0.57.254:36133 I0507 06:31:32.344322 6168 heartbeater.cc:426] Registering TS with master... I0507 06:31:32.344909 6168 heartbeater.cc:475] Master 127.0.57.254:36133 requested a full tablet report, sending... I0507 06:31:32.346184 5881 ts_manager.cc:159] Registered new tserver with Master: f82b67eec9cc44e0811eadb4fb49e7b3 (127.0.57.194:42507) I0507 06:31:32.347316 5881 master_service.cc:273] Signed X509 certificate for tserver {username='slave'} at 127.0.57.194:52423 WARNING: Logging before InitGoogleLogging() is written to STDERR W0507 06:31:32.512238 6171 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0507 06:31:32.512485 6171 flags.cc:404] Enabled unsafe flag: --never_fsync=true W0507 06:31:32.514704 6171 flags.cc:404] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true W0507 06:31:32.514761 6171 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W0507 06:31:32.514914 6171 flags.cc:404] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.57.195 I0507 06:31:32.519210 6171 tablet_server_main.cc:78] Tablet server non-default flags: --consensus_rpc_timeout_ms=5000 --follower_unavailable_considered_failed_sec=5 --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.TabletServerIsGoneAndBack.1557210500371863-231/raft_consensus-itest-cluster/ts-2/data --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.TabletServerIsGoneAndBack.1557210500371863-231/raft_consensus-itest-cluster/ts-2/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.0.57.195:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.TabletServerIsGoneAndBack.1557210500371863-231/raft_consensus-itest-cluster/ts-2/data/info.pb --webserver_interface=127.0.57.195 --webserver_port=0 --tablet_copy_download_file_inject_latency_ms=180000 --tserver_master_addrs=127.0.57.254:36133 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.0.57.195 --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.TabletServerIsGoneAndBack.1557210500371863-231/raft_consensus-itest-cluster/ts-2/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.10.0-cdh6.x-SNAPSHOT revision 7d0ee26697926e87af9ef395008799577f6ab43f build type FASTDEBUG built by jenkins at 06 May 2019 22:44:21 PST on quasar-knmpfx-1.vpc.cloudera.com build id 322017 ASAN enabled I0507 06:31:32.527372 6171 system_ntp.cc:202] NTP initialized. Skew: 500ppm Current error: 120516us I0507 06:31:32.528096 6171 fs_manager.cc:263] Metadata directory not provided I0507 06:31:32.528168 6171 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0507 06:31:32.528493 6171 server_base.cc:433] Could not load existing FS layout: Not found: could not find a healthy instance file I0507 06:31:32.528582 6171 server_base.cc:434] Attempting to create new FS layout instead I0507 06:31:32.531121 6171 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.TabletServerIsGoneAndBack.1557210500371863-231/raft_consensus-itest-cluster/ts-2/data/instance: uuid: "c4b5cf02494a4c9991316895da73ca72" format_stamp: "Formatted at 2019-05-07 06:31:32 on dist-test-slave-dist-test-slave-rqp6" I0507 06:31:32.531726 6171 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.TabletServerIsGoneAndBack.1557210500371863-231/raft_consensus-itest-cluster/ts-2/wal/instance: uuid: "c4b5cf02494a4c9991316895da73ca72" format_stamp: "Formatted at 2019-05-07 06:31:32 on dist-test-slave-dist-test-slave-rqp6" I0507 06:31:32.538568 6171 fs_manager.cc:503] Time spent creating directory manager: real 0.007s user 0.007s sys 0.002s I0507 06:31:32.539731 6171 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0507 06:31:32.540426 6171 file_cache.cc:466] Constructed file cache lbm with capacity 419430 I0507 06:31:32.543654 6171 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s I0507 06:31:32.543769 6171 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.TabletServerIsGoneAndBack.1557210500371863-231/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.TabletServerIsGoneAndBack.1557210500371863-231/raft_consensus-itest-cluster/ts-2/wal uuid: "c4b5cf02494a4c9991316895da73ca72" format_stamp: "Formatted at 2019-05-07 06:31:32 on dist-test-slave-dist-test-slave-rqp6" I0507 06:31:32.543948 6171 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.TabletServerIsGoneAndBack.1557210500371863-231/raft_consensus-itest-cluster/ts-2/wal metadata directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.TabletServerIsGoneAndBack.1557210500371863-231/raft_consensus-itest-cluster/ts-2/wal 1 data directories: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.TabletServerIsGoneAndBack.1557210500371863-231/raft_consensus-itest-cluster/ts-2/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0507 06:31:32.585708 6171 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0507 06:31:32.587666 6171 ts_tablet_manager.cc:356] Loaded tablet metadata (0 total tablets, 0 live tablets) I0507 06:31:32.587756 6171 ts_tablet_manager.cc:376] Registered 0 tablets I0507 06:31:32.614650 6171 rpc_server.cc:205] RPC server started. Bound to: 127.0.57.195:35959 I0507 06:31:32.615942 6171 webserver.cc:308] Webserver started at http://127.0.57.195:33893/ using document root and password file I0507 06:31:32.616717 6171 server_base.cc:608] Dumped server information to /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.TabletServerIsGoneAndBack.1557210500371863-231/raft_consensus-itest-cluster/ts-2/data/info.pb I0507 06:31:32.621480 231 external_mini_cluster.cc:995] Started /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver as pid 6171 I0507 06:31:32.622161 231 external_mini_cluster.cc:933] Running /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.TabletServerIsGoneAndBack.1557210500371863-231/raft_consensus-itest-cluster/ts-3/wal --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.TabletServerIsGoneAndBack.1557210500371863-231/raft_consensus-itest-cluster/ts-3/data --block_manager=log --rpc_bind_addresses=127.0.57.196:0 --local_ip_for_outbound_sockets=127.0.57.196 --webserver_interface=127.0.57.196 --webserver_port=0 --tserver_master_addrs=127.0.57.254:36133 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.TabletServerIsGoneAndBack.1557210500371863-231/raft_consensus-itest-cluster/ts-3/logs --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.TabletServerIsGoneAndBack.1557210500371863-231/raft_consensus-itest-cluster/ts-3/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=true --follower_unavailable_considered_failed_sec=5 --tablet_copy_download_file_inject_latency_ms=180000 --consensus_rpc_timeout_ms=5000 with env {} I0507 06:31:32.631285 6290 heartbeater.cc:345] Connected to a master server at 127.0.57.254:36133 I0507 06:31:32.631506 6290 heartbeater.cc:426] Registering TS with master... I0507 06:31:32.632097 6290 heartbeater.cc:475] Master 127.0.57.254:36133 requested a full tablet report, sending... I0507 06:31:32.633505 5881 ts_manager.cc:159] Registered new tserver with Master: c4b5cf02494a4c9991316895da73ca72 (127.0.57.195:35959) I0507 06:31:32.634948 5881 master_service.cc:273] Signed X509 certificate for tserver {username='slave'} at 127.0.57.195:50241 WARNING: Logging before InitGoogleLogging() is written to STDERR W0507 06:31:32.798409 6293 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0507 06:31:32.798652 6293 flags.cc:404] Enabled unsafe flag: --never_fsync=true W0507 06:31:32.800859 6293 flags.cc:404] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true W0507 06:31:32.800921 6293 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W0507 06:31:32.801021 6293 flags.cc:404] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.57.196 I0507 06:31:32.805181 6293 tablet_server_main.cc:78] Tablet server non-default flags: --consensus_rpc_timeout_ms=5000 --follower_unavailable_considered_failed_sec=5 --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.TabletServerIsGoneAndBack.1557210500371863-231/raft_consensus-itest-cluster/ts-3/data --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.TabletServerIsGoneAndBack.1557210500371863-231/raft_consensus-itest-cluster/ts-3/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.0.57.196:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.TabletServerIsGoneAndBack.1557210500371863-231/raft_consensus-itest-cluster/ts-3/data/info.pb --webserver_interface=127.0.57.196 --webserver_port=0 --tablet_copy_download_file_inject_latency_ms=180000 --tserver_master_addrs=127.0.57.254:36133 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.0.57.196 --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.TabletServerIsGoneAndBack.1557210500371863-231/raft_consensus-itest-cluster/ts-3/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.10.0-cdh6.x-SNAPSHOT revision 7d0ee26697926e87af9ef395008799577f6ab43f build type FASTDEBUG built by jenkins at 06 May 2019 22:44:21 PST on quasar-knmpfx-1.vpc.cloudera.com build id 322017 ASAN enabled I0507 06:31:32.813565 6293 system_ntp.cc:202] NTP initialized. Skew: 500ppm Current error: 120516us I0507 06:31:32.814301 6293 fs_manager.cc:263] Metadata directory not provided I0507 06:31:32.814379 6293 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0507 06:31:32.814682 6293 server_base.cc:433] Could not load existing FS layout: Not found: could not find a healthy instance file I0507 06:31:32.814752 6293 server_base.cc:434] Attempting to create new FS layout instead I0507 06:31:32.817287 6293 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.TabletServerIsGoneAndBack.1557210500371863-231/raft_consensus-itest-cluster/ts-3/data/instance: uuid: "b0a8e43040e44074b6b1aa9fe8f42e61" format_stamp: "Formatted at 2019-05-07 06:31:32 on dist-test-slave-dist-test-slave-rqp6" I0507 06:31:32.817909 6293 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.TabletServerIsGoneAndBack.1557210500371863-231/raft_consensus-itest-cluster/ts-3/wal/instance: uuid: "b0a8e43040e44074b6b1aa9fe8f42e61" format_stamp: "Formatted at 2019-05-07 06:31:32 on dist-test-slave-dist-test-slave-rqp6" I0507 06:31:32.824872 6293 fs_manager.cc:503] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s I0507 06:31:32.825986 6293 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0507 06:31:32.826618 6293 file_cache.cc:466] Constructed file cache lbm with capacity 419430 I0507 06:31:32.829916 6293 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s I0507 06:31:32.830015 6293 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.TabletServerIsGoneAndBack.1557210500371863-231/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.TabletServerIsGoneAndBack.1557210500371863-231/raft_consensus-itest-cluster/ts-3/wal uuid: "b0a8e43040e44074b6b1aa9fe8f42e61" format_stamp: "Formatted at 2019-05-07 06:31:32 on dist-test-slave-dist-test-slave-rqp6" I0507 06:31:32.830193 6293 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.TabletServerIsGoneAndBack.1557210500371863-231/raft_consensus-itest-cluster/ts-3/wal metadata directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.TabletServerIsGoneAndBack.1557210500371863-231/raft_consensus-itest-cluster/ts-3/wal 1 data directories: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.TabletServerIsGoneAndBack.1557210500371863-231/raft_consensus-itest-cluster/ts-3/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0507 06:31:32.876055 6293 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0507 06:31:32.878017 6293 ts_tablet_manager.cc:356] Loaded tablet metadata (0 total tablets, 0 live tablets) I0507 06:31:32.878096 6293 ts_tablet_manager.cc:376] Registered 0 tablets I0507 06:31:32.906002 6293 rpc_server.cc:205] RPC server started. Bound to: 127.0.57.196:36957 I0507 06:31:32.907297 6293 webserver.cc:308] Webserver started at http://127.0.57.196:39891/ using document root and password file I0507 06:31:32.908043 6293 server_base.cc:608] Dumped server information to /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.TabletServerIsGoneAndBack.1557210500371863-231/raft_consensus-itest-cluster/ts-3/data/info.pb I0507 06:31:32.911186 231 external_mini_cluster.cc:995] Started /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver as pid 6293 I0507 06:31:32.923593 6412 heartbeater.cc:345] Connected to a master server at 127.0.57.254:36133 I0507 06:31:32.923816 6412 heartbeater.cc:426] Registering TS with master... I0507 06:31:32.924368 6412 heartbeater.cc:475] Master 127.0.57.254:36133 requested a full tablet report, sending... I0507 06:31:32.925559 5881 ts_manager.cc:159] Registered new tserver with Master: b0a8e43040e44074b6b1aa9fe8f42e61 (127.0.57.196:36957) I0507 06:31:32.926515 5881 master_service.cc:273] Signed X509 certificate for tserver {username='slave'} at 127.0.57.196:34529 I0507 06:31:32.927531 231 external_mini_cluster.cc:623] 4 TS(s) registered with all masters I0507 06:31:32.939707 5881 catalog_manager.cc:1415] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49550: name: "TestTable" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 3 split_rows_range_bounds { } partition_schema { range_schema { columns { name: "key" } } } I0507 06:31:32.980355 6105 tablet_service.cc:1016] Processing CreateTablet for tablet 514b7d59d98b42f2a5aa360e82481c1c (table=TestTable [id=4f0b1c008b1d47df9810af186bfc6aec]), partition=RANGE (key) PARTITION UNBOUNDED I0507 06:31:32.980613 6227 tablet_service.cc:1016] Processing CreateTablet for tablet 514b7d59d98b42f2a5aa360e82481c1c (table=TestTable [id=4f0b1c008b1d47df9810af186bfc6aec]), partition=RANGE (key) PARTITION UNBOUNDED I0507 06:31:32.981187 6349 tablet_service.cc:1016] Processing CreateTablet for tablet 514b7d59d98b42f2a5aa360e82481c1c (table=TestTable [id=4f0b1c008b1d47df9810af186bfc6aec]), partition=RANGE (key) PARTITION UNBOUNDED I0507 06:31:32.981581 6105 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet 514b7d59d98b42f2a5aa360e82481c1c. 1 dirs total, 0 dirs full, 0 dirs failed I0507 06:31:32.981720 6227 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet 514b7d59d98b42f2a5aa360e82481c1c. 1 dirs total, 0 dirs full, 0 dirs failed I0507 06:31:32.982034 6349 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet 514b7d59d98b42f2a5aa360e82481c1c. 1 dirs total, 0 dirs full, 0 dirs failed I0507 06:31:32.996424 6432 tablet_bootstrap.cc:439] T 514b7d59d98b42f2a5aa360e82481c1c P b0a8e43040e44074b6b1aa9fe8f42e61: Bootstrap starting. I0507 06:31:32.997201 6433 tablet_bootstrap.cc:439] T 514b7d59d98b42f2a5aa360e82481c1c P f82b67eec9cc44e0811eadb4fb49e7b3: Bootstrap starting. I0507 06:31:32.997927 6434 tablet_bootstrap.cc:439] T 514b7d59d98b42f2a5aa360e82481c1c P c4b5cf02494a4c9991316895da73ca72: Bootstrap starting. I0507 06:31:33.002682 6433 tablet_bootstrap.cc:590] T 514b7d59d98b42f2a5aa360e82481c1c P f82b67eec9cc44e0811eadb4fb49e7b3: No blocks or log segments found. Creating new log. I0507 06:31:33.003651 6433 log.cc:525] T 514b7d59d98b42f2a5aa360e82481c1c P f82b67eec9cc44e0811eadb4fb49e7b3: Log is configured to *not* fsync() on all Append() calls I0507 06:31:33.006655 6432 tablet_bootstrap.cc:590] T 514b7d59d98b42f2a5aa360e82481c1c P b0a8e43040e44074b6b1aa9fe8f42e61: No blocks or log segments found. Creating new log. I0507 06:31:33.007601 6432 log.cc:525] T 514b7d59d98b42f2a5aa360e82481c1c P b0a8e43040e44074b6b1aa9fe8f42e61: Log is configured to *not* fsync() on all Append() calls I0507 06:31:33.008101 6434 tablet_bootstrap.cc:590] T 514b7d59d98b42f2a5aa360e82481c1c P c4b5cf02494a4c9991316895da73ca72: No blocks or log segments found. Creating new log. I0507 06:31:33.009634 6434 log.cc:525] T 514b7d59d98b42f2a5aa360e82481c1c P c4b5cf02494a4c9991316895da73ca72: Log is configured to *not* fsync() on all Append() calls I0507 06:31:33.010183 6433 tablet_bootstrap.cc:439] T 514b7d59d98b42f2a5aa360e82481c1c P f82b67eec9cc44e0811eadb4fb49e7b3: No bootstrap required, opened a new log I0507 06:31:33.010519 6433 ts_tablet_manager.cc:1053] T 514b7d59d98b42f2a5aa360e82481c1c P f82b67eec9cc44e0811eadb4fb49e7b3: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.000s I0507 06:31:33.011288 6432 tablet_bootstrap.cc:439] T 514b7d59d98b42f2a5aa360e82481c1c P b0a8e43040e44074b6b1aa9fe8f42e61: No bootstrap required, opened a new log I0507 06:31:33.011617 6434 tablet_bootstrap.cc:439] T 514b7d59d98b42f2a5aa360e82481c1c P c4b5cf02494a4c9991316895da73ca72: No bootstrap required, opened a new log I0507 06:31:33.011648 6432 ts_tablet_manager.cc:1053] T 514b7d59d98b42f2a5aa360e82481c1c P b0a8e43040e44074b6b1aa9fe8f42e61: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.000s I0507 06:31:33.011894 6434 ts_tablet_manager.cc:1053] T 514b7d59d98b42f2a5aa360e82481c1c P c4b5cf02494a4c9991316895da73ca72: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.002s I0507 06:31:33.016364 6433 raft_consensus.cc:340] T 514b7d59d98b42f2a5aa360e82481c1c P f82b67eec9cc44e0811eadb4fb49e7b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4b5cf02494a4c9991316895da73ca72" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 35959 } } peers { permanent_uuid: "b0a8e43040e44074b6b1aa9fe8f42e61" member_type: VOTER last_known_addr { host: "127.0.57.196" port: 36957 } } peers { permanent_uuid: "f82b67eec9cc44e0811eadb4fb49e7b3" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 42507 } } I0507 06:31:33.016773 6433 raft_consensus.cc:366] T 514b7d59d98b42f2a5aa360e82481c1c P f82b67eec9cc44e0811eadb4fb49e7b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0507 06:31:33.016857 6433 raft_consensus.cc:705] T 514b7d59d98b42f2a5aa360e82481c1c P f82b67eec9cc44e0811eadb4fb49e7b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f82b67eec9cc44e0811eadb4fb49e7b3, State: Initialized, Role: FOLLOWER I0507 06:31:33.017383 6433 consensus_queue.cc:229] T 514b7d59d98b42f2a5aa360e82481c1c P f82b67eec9cc44e0811eadb4fb49e7b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4b5cf02494a4c9991316895da73ca72" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 35959 } } peers { permanent_uuid: "b0a8e43040e44074b6b1aa9fe8f42e61" member_type: VOTER last_known_addr { host: "127.0.57.196" port: 36957 } } peers { permanent_uuid: "f82b67eec9cc44e0811eadb4fb49e7b3" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 42507 } } I0507 06:31:33.018255 6168 heartbeater.cc:467] Master 127.0.57.254:36133 was elected leader, sending a full tablet report... I0507 06:31:33.020148 6432 raft_consensus.cc:340] T 514b7d59d98b42f2a5aa360e82481c1c P b0a8e43040e44074b6b1aa9fe8f42e61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4b5cf02494a4c9991316895da73ca72" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 35959 } } peers { permanent_uuid: "b0a8e43040e44074b6b1aa9fe8f42e61" member_type: VOTER last_known_addr { host: "127.0.57.196" port: 36957 } } peers { permanent_uuid: "f82b67eec9cc44e0811eadb4fb49e7b3" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 42507 } } I0507 06:31:33.020328 6434 raft_consensus.cc:340] T 514b7d59d98b42f2a5aa360e82481c1c P c4b5cf02494a4c9991316895da73ca72 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4b5cf02494a4c9991316895da73ca72" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 35959 } } peers { permanent_uuid: "b0a8e43040e44074b6b1aa9fe8f42e61" member_type: VOTER last_known_addr { host: "127.0.57.196" port: 36957 } } peers { permanent_uuid: "f82b67eec9cc44e0811eadb4fb49e7b3" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 42507 } } I0507 06:31:33.020794 6432 raft_consensus.cc:366] T 514b7d59d98b42f2a5aa360e82481c1c P b0a8e43040e44074b6b1aa9fe8f42e61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0507 06:31:33.020870 6432 raft_consensus.cc:705] T 514b7d59d98b42f2a5aa360e82481c1c P b0a8e43040e44074b6b1aa9fe8f42e61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0a8e43040e44074b6b1aa9fe8f42e61, State: Initialized, Role: FOLLOWER I0507 06:31:33.020915 6434 raft_consensus.cc:366] T 514b7d59d98b42f2a5aa360e82481c1c P c4b5cf02494a4c9991316895da73ca72 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0507 06:31:33.020998 6434 raft_consensus.cc:705] T 514b7d59d98b42f2a5aa360e82481c1c P c4b5cf02494a4c9991316895da73ca72 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4b5cf02494a4c9991316895da73ca72, State: Initialized, Role: FOLLOWER I0507 06:31:33.021427 6433 ts_tablet_manager.cc:1081] T 514b7d59d98b42f2a5aa360e82481c1c P f82b67eec9cc44e0811eadb4fb49e7b3: Time spent starting tablet: real 0.011s user 0.002s sys 0.006s I0507 06:31:33.021523 6432 consensus_queue.cc:229] T 514b7d59d98b42f2a5aa360e82481c1c P b0a8e43040e44074b6b1aa9fe8f42e61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4b5cf02494a4c9991316895da73ca72" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 35959 } } peers { permanent_uuid: "b0a8e43040e44074b6b1aa9fe8f42e61" member_type: VOTER last_known_addr { host: "127.0.57.196" port: 36957 } } peers { permanent_uuid: "f82b67eec9cc44e0811eadb4fb49e7b3" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 42507 } } I0507 06:31:33.021523 6434 consensus_queue.cc:229] T 514b7d59d98b42f2a5aa360e82481c1c P c4b5cf02494a4c9991316895da73ca72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4b5cf02494a4c9991316895da73ca72" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 35959 } } peers { permanent_uuid: "b0a8e43040e44074b6b1aa9fe8f42e61" member_type: VOTER last_known_addr { host: "127.0.57.196" port: 36957 } } peers { permanent_uuid: "f82b67eec9cc44e0811eadb4fb49e7b3" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 42507 } } I0507 06:31:33.022572 6290 heartbeater.cc:467] Master 127.0.57.254:36133 was elected leader, sending a full tablet report... I0507 06:31:33.022883 6412 heartbeater.cc:467] Master 127.0.57.254:36133 was elected leader, sending a full tablet report... I0507 06:31:33.023849 6434 ts_tablet_manager.cc:1081] T 514b7d59d98b42f2a5aa360e82481c1c P c4b5cf02494a4c9991316895da73ca72: Time spent starting tablet: real 0.012s user 0.004s sys 0.007s I0507 06:31:33.024291 6432 ts_tablet_manager.cc:1081] T 514b7d59d98b42f2a5aa360e82481c1c P b0a8e43040e44074b6b1aa9fe8f42e61: Time spent starting tablet: real 0.013s user 0.008s sys 0.004s I0507 06:31:33.078580 6045 heartbeater.cc:467] Master 127.0.57.254:36133 was elected leader, sending a full tablet report... W0507 06:31:33.082442 6169 tablet.cc:1786] T 514b7d59d98b42f2a5aa360e82481c1c P f82b67eec9cc44e0811eadb4fb49e7b3: Can't schedule compaction. Clean time has not been advanced past its initial value. W0507 06:31:33.118819 6291 tablet.cc:1786] T 514b7d59d98b42f2a5aa360e82481c1c P c4b5cf02494a4c9991316895da73ca72: Can't schedule compaction. Clean time has not been advanced past its initial value. I0507 06:31:33.153281 6438 raft_consensus.cc:472] T 514b7d59d98b42f2a5aa360e82481c1c P f82b67eec9cc44e0811eadb4fb49e7b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0507 06:31:33.153594 6438 raft_consensus.cc:494] T 514b7d59d98b42f2a5aa360e82481c1c P f82b67eec9cc44e0811eadb4fb49e7b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4b5cf02494a4c9991316895da73ca72" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 35959 } } peers { permanent_uuid: "b0a8e43040e44074b6b1aa9fe8f42e61" member_type: VOTER last_known_addr { host: "127.0.57.196" port: 36957 } } peers { permanent_uuid: "f82b67eec9cc44e0811eadb4fb49e7b3" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 42507 } } I0507 06:31:33.155227 6438 leader_election.cc:296] T 514b7d59d98b42f2a5aa360e82481c1c P f82b67eec9cc44e0811eadb4fb49e7b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c4b5cf02494a4c9991316895da73ca72 (127.0.57.195:35959), b0a8e43040e44074b6b1aa9fe8f42e61 (127.0.57.196:36957) W0507 06:31:33.159293 6413 tablet.cc:1786] T 514b7d59d98b42f2a5aa360e82481c1c P b0a8e43040e44074b6b1aa9fe8f42e61: Can't schedule compaction. Clean time has not been advanced past its initial value. I0507 06:31:33.163185 6247 tablet_service.cc:1286] Received RequestConsensusVote() RPC: tablet_id: "514b7d59d98b42f2a5aa360e82481c1c" candidate_uuid: "f82b67eec9cc44e0811eadb4fb49e7b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4b5cf02494a4c9991316895da73ca72" is_pre_election: true I0507 06:31:33.163185 6369 tablet_service.cc:1286] Received RequestConsensusVote() RPC: tablet_id: "514b7d59d98b42f2a5aa360e82481c1c" candidate_uuid: "f82b67eec9cc44e0811eadb4fb49e7b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0a8e43040e44074b6b1aa9fe8f42e61" is_pre_election: true I0507 06:31:33.163632 6247 raft_consensus.cc:2346] T 514b7d59d98b42f2a5aa360e82481c1c P c4b5cf02494a4c9991316895da73ca72 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f82b67eec9cc44e0811eadb4fb49e7b3 in term 0. I0507 06:31:33.163632 6369 raft_consensus.cc:2346] T 514b7d59d98b42f2a5aa360e82481c1c P b0a8e43040e44074b6b1aa9fe8f42e61 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f82b67eec9cc44e0811eadb4fb49e7b3 in term 0. I0507 06:31:33.164484 6057 leader_election.cc:310] T 514b7d59d98b42f2a5aa360e82481c1c P f82b67eec9cc44e0811eadb4fb49e7b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c4b5cf02494a4c9991316895da73ca72, f82b67eec9cc44e0811eadb4fb49e7b3; no voters: I0507 06:31:33.165012 6438 raft_consensus.cc:2640] T 514b7d59d98b42f2a5aa360e82481c1c P f82b67eec9cc44e0811eadb4fb49e7b3 [term 0 FOLLOWER]: Leader pre-election won for term 1 I0507 06:31:33.165132 6438 raft_consensus.cc:472] T 514b7d59d98b42f2a5aa360e82481c1c P f82b67eec9cc44e0811eadb4fb49e7b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0507 06:31:33.165228 6438 raft_consensus.cc:2886] T 514b7d59d98b42f2a5aa360e82481c1c P f82b67eec9cc44e0811eadb4fb49e7b3 [term 0 FOLLOWER]: Advancing to term 1 I0507 06:31:33.167248 6438 raft_consensus.cc:494] T 514b7d59d98b42f2a5aa360e82481c1c P f82b67eec9cc44e0811eadb4fb49e7b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4b5cf02494a4c9991316895da73ca72" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 35959 } } peers { permanent_uuid: "b0a8e43040e44074b6b1aa9fe8f42e61" member_type: VOTER last_known_addr { host: "127.0.57.196" port: 36957 } } peers { permanent_uuid: "f82b67eec9cc44e0811eadb4fb49e7b3" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 42507 } } I0507 06:31:33.167845 6438 leader_election.cc:296] T 514b7d59d98b42f2a5aa360e82481c1c P f82b67eec9cc44e0811eadb4fb49e7b3 [CANDIDATE]: Term 1 election: Requested vote from peers c4b5cf02494a4c9991316895da73ca72 (127.0.57.195:35959), b0a8e43040e44074b6b1aa9fe8f42e61 (127.0.57.196:36957) I0507 06:31:33.168308 6247 tablet_service.cc:1286] Received RequestConsensusVote() RPC: tablet_id: "514b7d59d98b42f2a5aa360e82481c1c" candidate_uuid: "f82b67eec9cc44e0811eadb4fb49e7b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4b5cf02494a4c9991316895da73ca72" I0507 06:31:33.168493 6247 raft_consensus.cc:2886] T 514b7d59d98b42f2a5aa360e82481c1c P c4b5cf02494a4c9991316895da73ca72 [term 0 FOLLOWER]: Advancing to term 1 I0507 06:31:33.168599 6369 tablet_service.cc:1286] Received RequestConsensusVote() RPC: tablet_id: "514b7d59d98b42f2a5aa360e82481c1c" candidate_uuid: "f82b67eec9cc44e0811eadb4fb49e7b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0a8e43040e44074b6b1aa9fe8f42e61" I0507 06:31:33.168800 6369 raft_consensus.cc:2886] T 514b7d59d98b42f2a5aa360e82481c1c P b0a8e43040e44074b6b1aa9fe8f42e61 [term 0 FOLLOWER]: Advancing to term 1 I0507 06:31:33.170441 6247 raft_consensus.cc:2346] T 514b7d59d98b42f2a5aa360e82481c1c P c4b5cf02494a4c9991316895da73ca72 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f82b67eec9cc44e0811eadb4fb49e7b3 in term 1. I0507 06:31:33.170799 6369 raft_consensus.cc:2346] T 514b7d59d98b42f2a5aa360e82481c1c P b0a8e43040e44074b6b1aa9fe8f42e61 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f82b67eec9cc44e0811eadb4fb49e7b3 in term 1. I0507 06:31:33.170819 6057 leader_election.cc:310] T 514b7d59d98b42f2a5aa360e82481c1c P f82b67eec9cc44e0811eadb4fb49e7b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c4b5cf02494a4c9991316895da73ca72, f82b67eec9cc44e0811eadb4fb49e7b3; no voters: I0507 06:31:33.171063 6438 raft_consensus.cc:2640] T 514b7d59d98b42f2a5aa360e82481c1c P f82b67eec9cc44e0811eadb4fb49e7b3 [term 1 FOLLOWER]: Leader election won for term 1 I0507 06:31:33.171494 6438 raft_consensus.cc:667] T 514b7d59d98b42f2a5aa360e82481c1c P f82b67eec9cc44e0811eadb4fb49e7b3 [term 1 LEADER]: Becoming Leader. State: Replica: f82b67eec9cc44e0811eadb4fb49e7b3, State: Running, Role: LEADER I0507 06:31:33.171946 6438 consensus_queue.cc:206] T 514b7d59d98b42f2a5aa360e82481c1c P f82b67eec9cc44e0811eadb4fb49e7b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4b5cf02494a4c9991316895da73ca72" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 35959 } } peers { permanent_uuid: "b0a8e43040e44074b6b1aa9fe8f42e61" member_type: VOTER last_known_addr { host: "127.0.57.196" port: 36957 } } peers { permanent_uuid: "f82b67eec9cc44e0811eadb4fb49e7b3" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 42507 } } I0507 06:31:33.176290 5880 catalog_manager.cc:3988] T 514b7d59d98b42f2a5aa360e82481c1c P f82b67eec9cc44e0811eadb4fb49e7b3 reported cstate change: term changed from 0 to 1, leader changed from to f82b67eec9cc44e0811eadb4fb49e7b3 (127.0.57.194). New cstate: current_term: 1 leader_uuid: "f82b67eec9cc44e0811eadb4fb49e7b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4b5cf02494a4c9991316895da73ca72" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 35959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b0a8e43040e44074b6b1aa9fe8f42e61" member_type: VOTER last_known_addr { host: "127.0.57.196" port: 36957 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f82b67eec9cc44e0811eadb4fb49e7b3" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 42507 } health_report { overall_health: HEALTHY } } } I0507 06:31:33.211854 231 external_mini_cluster.cc:623] 4 TS(s) registered with all masters I0507 06:31:33.213361 231 ts_itest-base.cc:232] Waiting for 0 tablets on tserver bc60338af9984198a753a6394960edcd to finish bootstrapping I0507 06:31:33.223974 231 ts_itest-base.cc:232] Waiting for 1 tablets on tserver f82b67eec9cc44e0811eadb4fb49e7b3 to finish bootstrapping I0507 06:31:33.230500 231 ts_itest-base.cc:232] Waiting for 1 tablets on tserver c4b5cf02494a4c9991316895da73ca72 to finish bootstrapping I0507 06:31:33.237177 231 ts_itest-base.cc:232] Waiting for 1 tablets on tserver b0a8e43040e44074b6b1aa9fe8f42e61 to finish bootstrapping I0507 06:31:33.244021 231 test_util.cc:212] Using random seed: 1580602441 I0507 06:31:33.259943 231 test_workload.cc:306] TestWorkload: Skipping table creation because table TestTable already exists I0507 06:31:33.306288 6247 raft_consensus.cc:1181] T 514b7d59d98b42f2a5aa360e82481c1c P c4b5cf02494a4c9991316895da73ca72 [term 1 FOLLOWER]: Refusing update from remote peer f82b67eec9cc44e0811eadb4fb49e7b3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0507 06:31:33.306447 6369 raft_consensus.cc:1181] T 514b7d59d98b42f2a5aa360e82481c1c P b0a8e43040e44074b6b1aa9fe8f42e61 [term 1 FOLLOWER]: Refusing update from remote peer f82b67eec9cc44e0811eadb4fb49e7b3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0507 06:31:33.307600 6443 consensus_queue.cc:984] T 514b7d59d98b42f2a5aa360e82481c1c P f82b67eec9cc44e0811eadb4fb49e7b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b0a8e43040e44074b6b1aa9fe8f42e61" member_type: VOTER last_known_addr { host: "127.0.57.196" port: 36957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0507 06:31:33.307956 6438 consensus_queue.cc:984] T 514b7d59d98b42f2a5aa360e82481c1c P f82b67eec9cc44e0811eadb4fb49e7b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4b5cf02494a4c9991316895da73ca72" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 35959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0507 06:31:33.327353 6456 mvcc.cc:203] Tried to move safe_time back from 6378334999771860992 to 6378334999235731456. Current Snapshot: MvccSnapshot[committed={T|T < 6378334999771860992}] I0507 06:31:33.372748 231 external_mini_cluster.cc:1172] Attempting to check leaks for /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver pid 6293 I0507 06:31:33.373827 6409 generic_service.cc:171] Checking for leaks (request via RPC) I0507 06:31:33.435750 231 external_mini_cluster.cc:1131] Killing /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver with pid 6293 W0507 06:31:33.897626 6058 consensus_peers.cc:457] T 514b7d59d98b42f2a5aa360e82481c1c P f82b67eec9cc44e0811eadb4fb49e7b3 -> Peer b0a8e43040e44074b6b1aa9fe8f42e61 (127.0.57.196:36957): Couldn't send request to peer b0a8e43040e44074b6b1aa9fe8f42e61. Status: Network error: Client connection negotiation failed: client connection to 127.0.57.196:36957: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry. W0507 06:31:36.317442 6058 consensus_peers.cc:457] T 514b7d59d98b42f2a5aa360e82481c1c P f82b67eec9cc44e0811eadb4fb49e7b3 -> Peer b0a8e43040e44074b6b1aa9fe8f42e61 (127.0.57.196:36957): Couldn't send request to peer b0a8e43040e44074b6b1aa9fe8f42e61. Status: Network error: Client connection negotiation failed: client connection to 127.0.57.196:36957: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry. I0507 06:31:38.470072 6477 consensus_queue.cc:544] T 514b7d59d98b42f2a5aa360e82481c1c P f82b67eec9cc44e0811eadb4fb49e7b3 [LEADER]: Leader has been unable to successfully communicate with peer b0a8e43040e44074b6b1aa9fe8f42e61 for more than 5 seconds (5.101s) I0507 06:31:38.472442 5873 catalog_manager.cc:3616] Sending ChangeConfig:ADD_PEER:NON_VOTER on tablet 514b7d59d98b42f2a5aa360e82481c1c (attempt 1) I0507 06:31:38.474783 6125 consensus_queue.cc:206] T 514b7d59d98b42f2a5aa360e82481c1c P f82b67eec9cc44e0811eadb4fb49e7b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c4b5cf02494a4c9991316895da73ca72" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 35959 } } peers { permanent_uuid: "b0a8e43040e44074b6b1aa9fe8f42e61" member_type: VOTER last_known_addr { host: "127.0.57.196" port: 36957 } } peers { permanent_uuid: "f82b67eec9cc44e0811eadb4fb49e7b3" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 42507 } } peers { permanent_uuid: "bc60338af9984198a753a6394960edcd" member_type: NON_VOTER last_known_addr { host: "127.0.57.193" port: 33675 } attrs { promote: true } } W0507 06:31:38.480425 6058 consensus_peers.cc:457] T 514b7d59d98b42f2a5aa360e82481c1c P f82b67eec9cc44e0811eadb4fb49e7b3 -> Peer b0a8e43040e44074b6b1aa9fe8f42e61 (127.0.57.196:36957): Couldn't send request to peer b0a8e43040e44074b6b1aa9fe8f42e61. Status: Network error: Client connection negotiation failed: client connection to 127.0.57.196:36957: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry. I0507 06:31:38.481113 6247 raft_consensus.cc:1181] T 514b7d59d98b42f2a5aa360e82481c1c P c4b5cf02494a4c9991316895da73ca72 [term 1 FOLLOWER]: Refusing update from remote peer f82b67eec9cc44e0811eadb4fb49e7b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch) I0507 06:31:38.481607 6477 consensus_queue.cc:984] T 514b7d59d98b42f2a5aa360e82481c1c P f82b67eec9cc44e0811eadb4fb49e7b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4b5cf02494a4c9991316895da73ca72" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 35959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s W0507 06:31:38.482722 6056 consensus_peers.cc:457] T 514b7d59d98b42f2a5aa360e82481c1c P f82b67eec9cc44e0811eadb4fb49e7b3 -> Peer bc60338af9984198a753a6394960edcd (127.0.57.193:33675): Couldn't send request to peer bc60338af9984198a753a6394960edcd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 514b7d59d98b42f2a5aa360e82481c1c. This is attempt 1: this message will repeat every 5th retry. I0507 06:31:38.484254 6481 raft_consensus.cc:2788] T 514b7d59d98b42f2a5aa360e82481c1c P f82b67eec9cc44e0811eadb4fb49e7b3 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index -1 to 10, NON_VOTER bc60338af9984198a753a6394960edcd (127.0.57.193) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c4b5cf02494a4c9991316895da73ca72" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 35959 } } peers { permanent_uuid: "b0a8e43040e44074b6b1aa9fe8f42e61" member_type: VOTER last_known_addr { host: "127.0.57.196" port: 36957 } } peers { permanent_uuid: "f82b67eec9cc44e0811eadb4fb49e7b3" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 42507 } } peers { permanent_uuid: "bc60338af9984198a753a6394960edcd" member_type: NON_VOTER last_known_addr { host: "127.0.57.193" port: 33675 } attrs { promote: true } } } I0507 06:31:38.484679 6247 raft_consensus.cc:2788] T 514b7d59d98b42f2a5aa360e82481c1c P c4b5cf02494a4c9991316895da73ca72 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index -1 to 10, NON_VOTER bc60338af9984198a753a6394960edcd (127.0.57.193) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c4b5cf02494a4c9991316895da73ca72" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 35959 } } peers { permanent_uuid: "b0a8e43040e44074b6b1aa9fe8f42e61" member_type: VOTER last_known_addr { host: "127.0.57.196" port: 36957 } } peers { permanent_uuid: "f82b67eec9cc44e0811eadb4fb49e7b3" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 42507 } } peers { permanent_uuid: "bc60338af9984198a753a6394960edcd" member_type: NON_VOTER last_known_addr { host: "127.0.57.193" port: 33675 } attrs { promote: true } } } I0507 06:31:38.487336 5864 catalog_manager.cc:3531] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 514b7d59d98b42f2a5aa360e82481c1c with cas_config_opid_index -1: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1) I0507 06:31:38.488356 5873 catalog_manager.cc:3988] T 514b7d59d98b42f2a5aa360e82481c1c P f82b67eec9cc44e0811eadb4fb49e7b3 reported cstate change: config changed from index -1 to 10, NON_VOTER bc60338af9984198a753a6394960edcd (127.0.57.193) added. New cstate: current_term: 1 leader_uuid: "f82b67eec9cc44e0811eadb4fb49e7b3" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c4b5cf02494a4c9991316895da73ca72" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 35959 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b0a8e43040e44074b6b1aa9fe8f42e61" member_type: VOTER last_known_addr { host: "127.0.57.196" port: 36957 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f82b67eec9cc44e0811eadb4fb49e7b3" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 42507 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bc60338af9984198a753a6394960edcd" member_type: NON_VOTER last_known_addr { host: "127.0.57.193" port: 33675 } attrs { promote: true } health_report { overall_health: UNKNOWN } } } I0507 06:31:38.512334 231 external_mini_cluster.cc:933] Running /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.TabletServerIsGoneAndBack.1557210500371863-231/raft_consensus-itest-cluster/ts-3/wal --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.TabletServerIsGoneAndBack.1557210500371863-231/raft_consensus-itest-cluster/ts-3/data --block_manager=log --rpc_bind_addresses=127.0.57.196:36957 --local_ip_for_outbound_sockets=127.0.57.196 --webserver_port=39891 --webserver_interface=127.0.57.196 --tserver_master_addrs=127.0.57.254:36133 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.TabletServerIsGoneAndBack.1557210500371863-231/raft_consensus-itest-cluster/ts-3/logs --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.TabletServerIsGoneAndBack.1557210500371863-231/raft_consensus-itest-cluster/ts-3/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=true --follower_unavailable_considered_failed_sec=5 --tablet_copy_download_file_inject_latency_ms=180000 --consensus_rpc_timeout_ms=5000 with env {} WARNING: Logging before InitGoogleLogging() is written to STDERR W0507 06:31:38.681588 6493 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0507 06:31:38.681844 6493 flags.cc:404] Enabled unsafe flag: --never_fsync=true W0507 06:31:38.683996 6493 flags.cc:404] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true W0507 06:31:38.684051 6493 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W0507 06:31:38.684156 6493 flags.cc:404] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.57.196 I0507 06:31:38.688541 6493 tablet_server_main.cc:78] Tablet server non-default flags: --consensus_rpc_timeout_ms=5000 --follower_unavailable_considered_failed_sec=5 --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.TabletServerIsGoneAndBack.1557210500371863-231/raft_consensus-itest-cluster/ts-3/data --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.TabletServerIsGoneAndBack.1557210500371863-231/raft_consensus-itest-cluster/ts-3/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.0.57.196:36957 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.TabletServerIsGoneAndBack.1557210500371863-231/raft_consensus-itest-cluster/ts-3/data/info.pb --webserver_interface=127.0.57.196 --webserver_port=39891 --tablet_copy_download_file_inject_latency_ms=180000 --tserver_master_addrs=127.0.57.254:36133 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.0.57.196 --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.TabletServerIsGoneAndBack.1557210500371863-231/raft_consensus-itest-cluster/ts-3/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.10.0-cdh6.x-SNAPSHOT revision 7d0ee26697926e87af9ef395008799577f6ab43f build type FASTDEBUG built by jenkins at 06 May 2019 22:44:21 PST on quasar-knmpfx-1.vpc.cloudera.com build id 322017 ASAN enabled I0507 06:31:38.696696 6493 system_ntp.cc:202] NTP initialized. Skew: 500ppm Current error: 123516us I0507 06:31:38.697448 6493 fs_manager.cc:263] Metadata directory not provided I0507 06:31:38.697528 6493 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0507 06:31:38.703873 6493 fs_manager.cc:399] Time spent opening directory manager: real 0.005s user 0.004s sys 0.002s I0507 06:31:38.704588 6493 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0507 06:31:38.705188 6493 file_cache.cc:466] Constructed file cache lbm with capacity 419430 I0507 06:31:38.708621 6493 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s I0507 06:31:38.708730 6493 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.TabletServerIsGoneAndBack.1557210500371863-231/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.TabletServerIsGoneAndBack.1557210500371863-231/raft_consensus-itest-cluster/ts-3/wal uuid: "b0a8e43040e44074b6b1aa9fe8f42e61" format_stamp: "Formatted at 2019-05-07 06:31:32 on dist-test-slave-dist-test-slave-rqp6" I0507 06:31:38.709728 6493 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.TabletServerIsGoneAndBack.1557210500371863-231/raft_consensus-itest-cluster/ts-3/wal metadata directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.TabletServerIsGoneAndBack.1557210500371863-231/raft_consensus-itest-cluster/ts-3/wal 1 data directories: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.TabletServerIsGoneAndBack.1557210500371863-231/raft_consensus-itest-cluster/ts-3/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0507 06:31:38.751152 6493 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0507 06:31:38.752995 6493 ts_tablet_manager.cc:344] Loading tablet metadata (0/1 complete) I0507 06:31:38.756795 6493 ts_tablet_manager.cc:356] Loaded tablet metadata (1 total tablets, 1 live tablets) I0507 06:31:38.756861 6493 ts_tablet_manager.cc:362] Registering tablets (0/1 complete) I0507 06:31:38.759656 6508 tablet_bootstrap.cc:439] T 514b7d59d98b42f2a5aa360e82481c1c P b0a8e43040e44074b6b1aa9fe8f42e61: Bootstrap starting. I0507 06:31:38.764629 6493 ts_tablet_manager.cc:376] Registered 1 tablets I0507 06:31:38.777380 6508 log.cc:525] T 514b7d59d98b42f2a5aa360e82481c1c P b0a8e43040e44074b6b1aa9fe8f42e61: Log is configured to *not* fsync() on all Append() calls I0507 06:31:38.804471 6493 rpc_server.cc:205] RPC server started. Bound to: 127.0.57.196:36957 I0507 06:31:38.805776 6493 webserver.cc:308] Webserver started at http://127.0.57.196:39891/ using document root and password file I0507 06:31:38.806473 6493 server_base.cc:608] Dumped server information to /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.TabletServerIsGoneAndBack.1557210500371863-231/raft_consensus-itest-cluster/ts-3/data/info.pb I0507 06:31:38.808686 6508 tablet_bootstrap.cc:439] T 514b7d59d98b42f2a5aa360e82481c1c P b0a8e43040e44074b6b1aa9fe8f42e61: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=7 ignored=0} inserts{seen=300 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 2 replicates I0507 06:31:38.809378 6508 tablet_bootstrap.cc:439] T 514b7d59d98b42f2a5aa360e82481c1c P b0a8e43040e44074b6b1aa9fe8f42e61: Bootstrap complete. I0507 06:31:38.810305 6508 ts_tablet_manager.cc:1053] T 514b7d59d98b42f2a5aa360e82481c1c P b0a8e43040e44074b6b1aa9fe8f42e61: Time spent bootstrapping tablet: real 0.051s user 0.035s sys 0.008s I0507 06:31:38.810863 231 external_mini_cluster.cc:995] Started /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver as pid 6493 I0507 06:31:38.821233 6508 raft_consensus.cc:340] T 514b7d59d98b42f2a5aa360e82481c1c P b0a8e43040e44074b6b1aa9fe8f42e61 [term 1 FOLLOWER]: Replica starting. Triggering 2 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4b5cf02494a4c9991316895da73ca72" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 35959 } } peers { permanent_uuid: "b0a8e43040e44074b6b1aa9fe8f42e61" member_type: VOTER last_known_addr { host: "127.0.57.196" port: 36957 } } peers { permanent_uuid: "f82b67eec9cc44e0811eadb4fb49e7b3" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 42507 } } I0507 06:31:38.825664 6614 heartbeater.cc:345] Connected to a master server at 127.0.57.254:36133 I0507 06:31:38.825901 6614 heartbeater.cc:426] Registering TS with master... I0507 06:31:38.826514 6614 heartbeater.cc:475] Master 127.0.57.254:36133 requested a full tablet report, sending... I0507 06:31:38.827476 6508 raft_consensus.cc:705] T 514b7d59d98b42f2a5aa360e82481c1c P b0a8e43040e44074b6b1aa9fe8f42e61 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0a8e43040e44074b6b1aa9fe8f42e61, State: Initialized, Role: FOLLOWER I0507 06:31:38.828112 6508 consensus_queue.cc:229] T 514b7d59d98b42f2a5aa360e82481c1c P b0a8e43040e44074b6b1aa9fe8f42e61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4b5cf02494a4c9991316895da73ca72" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 35959 } } peers { permanent_uuid: "b0a8e43040e44074b6b1aa9fe8f42e61" member_type: VOTER last_known_addr { host: "127.0.57.196" port: 36957 } } peers { permanent_uuid: "f82b67eec9cc44e0811eadb4fb49e7b3" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 42507 } } I0507 06:31:38.830375 6508 ts_tablet_manager.cc:1081] T 514b7d59d98b42f2a5aa360e82481c1c P b0a8e43040e44074b6b1aa9fe8f42e61: Time spent starting tablet: real 0.020s user 0.012s sys 0.002s I0507 06:31:38.830718 5880 ts_manager.cc:159] Re-registered known tserver with Master: b0a8e43040e44074b6b1aa9fe8f42e61 (127.0.57.196:36957) I0507 06:31:38.831856 5880 master_service.cc:273] Signed X509 certificate for tserver {username='slave'} at 127.0.57.196:59589 I0507 06:31:38.834139 6614 heartbeater.cc:467] Master 127.0.57.254:36133 was elected leader, sending a full tablet report... I0507 06:31:39.014483 6571 raft_consensus.cc:2788] T 514b7d59d98b42f2a5aa360e82481c1c P b0a8e43040e44074b6b1aa9fe8f42e61 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index -1 to 10, NON_VOTER bc60338af9984198a753a6394960edcd (127.0.57.193) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c4b5cf02494a4c9991316895da73ca72" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 35959 } } peers { permanent_uuid: "b0a8e43040e44074b6b1aa9fe8f42e61" member_type: VOTER last_known_addr { host: "127.0.57.196" port: 36957 } } peers { permanent_uuid: "f82b67eec9cc44e0811eadb4fb49e7b3" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 42507 } } peers { permanent_uuid: "bc60338af9984198a753a6394960edcd" member_type: NON_VOTER last_known_addr { host: "127.0.57.193" port: 33675 } attrs { promote: true } } } I0507 06:31:39.046241 6625 ts_tablet_manager.cc:676] T 514b7d59d98b42f2a5aa360e82481c1c P bc60338af9984198a753a6394960edcd: Initiating tablet copy from peer f82b67eec9cc44e0811eadb4fb49e7b3 (127.0.57.194:42507) I0507 06:31:39.046563 6625 tablet_copy_client.cc:240] T 514b7d59d98b42f2a5aa360e82481c1c P bc60338af9984198a753a6394960edcd: tablet copy: Beginning tablet copy session from remote peer at address 127.0.57.194:42507 I0507 06:31:39.053027 6145 tablet_copy_service.cc:135] P f82b67eec9cc44e0811eadb4fb49e7b3: Received BeginTabletCopySession request for tablet 514b7d59d98b42f2a5aa360e82481c1c from peer bc60338af9984198a753a6394960edcd ({username='slave'} at 127.0.57.193:54883) I0507 06:31:39.053202 6145 tablet_copy_service.cc:156] P f82b67eec9cc44e0811eadb4fb49e7b3: Beginning new tablet copy session on tablet 514b7d59d98b42f2a5aa360e82481c1c from peer bc60338af9984198a753a6394960edcd at {username='slave'} at 127.0.57.193:54883: session id = bc60338af9984198a753a6394960edcd-514b7d59d98b42f2a5aa360e82481c1c I0507 06:31:39.055553 6145 tablet_copy_source_session.cc:202] T 514b7d59d98b42f2a5aa360e82481c1c P f82b67eec9cc44e0811eadb4fb49e7b3: Tablet Copy: opened 0 blocks and 1 log segments I0507 06:31:39.059187 6625 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet 514b7d59d98b42f2a5aa360e82481c1c. 1 dirs total, 0 dirs full, 0 dirs failed I0507 06:31:39.068199 6625 tablet_copy_client.cc:575] T 514b7d59d98b42f2a5aa360e82481c1c P bc60338af9984198a753a6394960edcd: tablet copy: Starting download of 0 data blocks... I0507 06:31:39.068361 6625 tablet_copy_client.cc:538] T 514b7d59d98b42f2a5aa360e82481c1c P bc60338af9984198a753a6394960edcd: tablet copy: Starting download of 1 WAL segments... I0507 06:31:39.070094 6625 tablet_copy_client.cc:749] T 514b7d59d98b42f2a5aa360e82481c1c P bc60338af9984198a753a6394960edcd: tablet copy: Injecting latency into file download: 180000 I0507 06:31:39.460242 5880 catalog_manager.cc:3739] Sending ChangeConfig:REMOVE_PEER on tablet 514b7d59d98b42f2a5aa360e82481c1c (attempt 1) I0507 06:31:39.461582 6125 consensus_queue.cc:206] T 514b7d59d98b42f2a5aa360e82481c1c P f82b67eec9cc44e0811eadb4fb49e7b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c4b5cf02494a4c9991316895da73ca72" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 35959 } } peers { permanent_uuid: "b0a8e43040e44074b6b1aa9fe8f42e61" member_type: VOTER last_known_addr { host: "127.0.57.196" port: 36957 } } peers { permanent_uuid: "f82b67eec9cc44e0811eadb4fb49e7b3" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 42507 } } I0507 06:31:39.463429 6571 raft_consensus.cc:1181] T 514b7d59d98b42f2a5aa360e82481c1c P b0a8e43040e44074b6b1aa9fe8f42e61 [term 1 FOLLOWER]: Refusing update from remote peer f82b67eec9cc44e0811eadb4fb49e7b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch) I0507 06:31:39.463557 6247 raft_consensus.cc:1181] T 514b7d59d98b42f2a5aa360e82481c1c P c4b5cf02494a4c9991316895da73ca72 [term 1 FOLLOWER]: Refusing update from remote peer f82b67eec9cc44e0811eadb4fb49e7b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch) I0507 06:31:39.464100 6628 consensus_queue.cc:984] T 514b7d59d98b42f2a5aa360e82481c1c P f82b67eec9cc44e0811eadb4fb49e7b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b0a8e43040e44074b6b1aa9fe8f42e61" member_type: VOTER last_known_addr { host: "127.0.57.196" port: 36957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s I0507 06:31:39.464371 6481 consensus_queue.cc:984] T 514b7d59d98b42f2a5aa360e82481c1c P f82b67eec9cc44e0811eadb4fb49e7b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4b5cf02494a4c9991316895da73ca72" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 35959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s I0507 06:31:39.466831 6481 raft_consensus.cc:2788] T 514b7d59d98b42f2a5aa360e82481c1c P f82b67eec9cc44e0811eadb4fb49e7b3 [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER bc60338af9984198a753a6394960edcd (127.0.57.193) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c4b5cf02494a4c9991316895da73ca72" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 35959 } } peers { permanent_uuid: "b0a8e43040e44074b6b1aa9fe8f42e61" member_type: VOTER last_known_addr { host: "127.0.57.196" port: 36957 } } peers { permanent_uuid: "f82b67eec9cc44e0811eadb4fb49e7b3" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 42507 } } } I0507 06:31:39.467272 6247 raft_consensus.cc:2788] T 514b7d59d98b42f2a5aa360e82481c1c P c4b5cf02494a4c9991316895da73ca72 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER bc60338af9984198a753a6394960edcd (127.0.57.193) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c4b5cf02494a4c9991316895da73ca72" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 35959 } } peers { permanent_uuid: "b0a8e43040e44074b6b1aa9fe8f42e61" member_type: VOTER last_known_addr { host: "127.0.57.196" port: 36957 } } peers { permanent_uuid: "f82b67eec9cc44e0811eadb4fb49e7b3" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 42507 } } } I0507 06:31:39.469071 6571 raft_consensus.cc:2788] T 514b7d59d98b42f2a5aa360e82481c1c P b0a8e43040e44074b6b1aa9fe8f42e61 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER bc60338af9984198a753a6394960edcd (127.0.57.193) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c4b5cf02494a4c9991316895da73ca72" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 35959 } } peers { permanent_uuid: "b0a8e43040e44074b6b1aa9fe8f42e61" member_type: VOTER last_known_addr { host: "127.0.57.196" port: 36957 } } peers { permanent_uuid: "f82b67eec9cc44e0811eadb4fb49e7b3" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 42507 } } } I0507 06:31:39.469549 5864 catalog_manager.cc:3531] ChangeConfig:REMOVE_PEER RPC for tablet 514b7d59d98b42f2a5aa360e82481c1c with cas_config_opid_index 10: ChangeConfig:REMOVE_PEER succeeded (attempt 1) I0507 06:31:39.470333 5880 catalog_manager.cc:3988] T 514b7d59d98b42f2a5aa360e82481c1c P f82b67eec9cc44e0811eadb4fb49e7b3 reported cstate change: config changed from index 10 to 11, NON_VOTER bc60338af9984198a753a6394960edcd (127.0.57.193) evicted. New cstate: current_term: 1 leader_uuid: "f82b67eec9cc44e0811eadb4fb49e7b3" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c4b5cf02494a4c9991316895da73ca72" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 35959 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b0a8e43040e44074b6b1aa9fe8f42e61" member_type: VOTER last_known_addr { host: "127.0.57.196" port: 36957 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f82b67eec9cc44e0811eadb4fb49e7b3" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 42507 } health_report { overall_health: HEALTHY } } } I0507 06:31:39.474879 5880 catalog_manager.cc:3390] Sending DeleteTablet(TABLET_DATA_TOMBSTONED) for tablet 514b7d59d98b42f2a5aa360e82481c1c on bc60338af9984198a753a6394960edcd (127.0.57.193:33675) (TS bc60338af9984198a753a6394960edcd not found in new config with opid_index 11) I0507 06:31:39.480751 5982 tablet_service.cc:1057] Processing DeleteTablet for tablet 514b7d59d98b42f2a5aa360e82481c1c with delete_type TABLET_DATA_TOMBSTONED (TS bc60338af9984198a753a6394960edcd not found in new config with opid_index 11) from {username='slave'} at 127.0.0.1:50020 W0507 06:31:39.482306 5862 catalog_manager.cc:3360] TS bc60338af9984198a753a6394960edcd (127.0.57.193:33675): delete failed for tablet 514b7d59d98b42f2a5aa360e82481c1c with error code TABLET_NOT_RUNNING: Already present: State transition of tablet 514b7d59d98b42f2a5aa360e82481c1c already in progress: copying tablet I0507 06:31:39.482391 5862 catalog_manager.cc:3168] Scheduling retry of 514b7d59d98b42f2a5aa360e82481c1c Delete Tablet RPC for TS=bc60338af9984198a753a6394960edcd with a delay of 32 ms (attempt = 1) I0507 06:31:39.507709 231 external_mini_cluster.cc:1172] Attempting to check leaks for /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver pid 5915 I0507 06:31:39.509696 6042 generic_service.cc:171] Checking for leaks (request via RPC) I0507 06:31:39.515264 5862 catalog_manager.cc:3390] Sending DeleteTablet(TABLET_DATA_TOMBSTONED) for tablet 514b7d59d98b42f2a5aa360e82481c1c on bc60338af9984198a753a6394960edcd (127.0.57.193:33675) (TS bc60338af9984198a753a6394960edcd not found in new config with opid_index 11) I0507 06:31:39.553156 5982 tablet_service.cc:1057] Processing DeleteTablet for tablet 514b7d59d98b42f2a5aa360e82481c1c with delete_type TABLET_DATA_TOMBSTONED (TS bc60338af9984198a753a6394960edcd not found in new config with opid_index 11) from {username='slave'} at 127.0.0.1:50020 W0507 06:31:39.554275 5862 catalog_manager.cc:3360] TS bc60338af9984198a753a6394960edcd (127.0.57.193:33675): delete failed for tablet 514b7d59d98b42f2a5aa360e82481c1c with error code TABLET_NOT_RUNNING: Already present: State transition of tablet 514b7d59d98b42f2a5aa360e82481c1c already in progress: copying tablet I0507 06:31:39.554414 5862 catalog_manager.cc:3168] Scheduling retry of 514b7d59d98b42f2a5aa360e82481c1c Delete Tablet RPC for TS=bc60338af9984198a753a6394960edcd with a delay of 79 ms (attempt = 2) I0507 06:31:39.554461 231 external_mini_cluster.cc:1131] Killing /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver with pid 5915 I0507 06:31:39.566395 231 external_mini_cluster.cc:1172] Attempting to check leaks for /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver pid 6049 I0507 06:31:39.567359 6165 generic_service.cc:171] Checking for leaks (request via RPC) I0507 06:31:39.622387 231 external_mini_cluster.cc:1131] Killing /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver with pid 6049 I0507 06:31:39.634610 5862 catalog_manager.cc:3390] Sending DeleteTablet(TABLET_DATA_TOMBSTONED) for tablet 514b7d59d98b42f2a5aa360e82481c1c on bc60338af9984198a753a6394960edcd (127.0.57.193:33675) (TS bc60338af9984198a753a6394960edcd not found in new config with opid_index 11) I0507 06:31:39.635541 231 external_mini_cluster.cc:1172] Attempting to check leaks for /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver pid 6171 W0507 06:31:39.635689 5862 catalog_manager.cc:3132] TS bc60338af9984198a753a6394960edcd (127.0.57.193:33675): Delete Tablet RPC failed for tablet 514b7d59d98b42f2a5aa360e82481c1c: Network error: Client connection negotiation failed: client connection to 127.0.57.193:33675: connect: Connection refused (error 111) I0507 06:31:39.635804 5862 catalog_manager.cc:3168] Scheduling retry of 514b7d59d98b42f2a5aa360e82481c1c Delete Tablet RPC for TS=bc60338af9984198a753a6394960edcd with a delay of 112 ms (attempt = 3) I0507 06:31:39.636636 6287 generic_service.cc:171] Checking for leaks (request via RPC) I0507 06:31:39.689669 231 external_mini_cluster.cc:1131] Killing /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver with pid 6171 I0507 06:31:39.701370 231 external_mini_cluster.cc:1172] Attempting to check leaks for /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver pid 6493 I0507 06:31:39.707159 6611 generic_service.cc:171] Checking for leaks (request via RPC) I0507 06:31:39.748745 5862 catalog_manager.cc:3390] Sending DeleteTablet(TABLET_DATA_TOMBSTONED) for tablet 514b7d59d98b42f2a5aa360e82481c1c on bc60338af9984198a753a6394960edcd (127.0.57.193:33675) (TS bc60338af9984198a753a6394960edcd not found in new config with opid_index 11) W0507 06:31:39.749689 5862 catalog_manager.cc:3132] TS bc60338af9984198a753a6394960edcd (127.0.57.193:33675): Delete Tablet RPC failed for tablet 514b7d59d98b42f2a5aa360e82481c1c: Network error: Client connection negotiation failed: client connection to 127.0.57.193:33675: connect: Connection refused (error 111) I0507 06:31:39.749790 5862 catalog_manager.cc:3168] Scheduling retry of 514b7d59d98b42f2a5aa360e82481c1c Delete Tablet RPC for TS=bc60338af9984198a753a6394960edcd with a delay of 175 ms (attempt = 4) I0507 06:31:39.759896 231 external_mini_cluster.cc:1131] Killing /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver with pid 6493 I0507 06:31:39.770294 231 external_mini_cluster.cc:1172] Attempting to check leaks for /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-master pid 5855 I0507 06:31:39.771390 5911 generic_service.cc:171] Checking for leaks (request via RPC) I0507 06:31:39.851361 231 external_mini_cluster.cc:1131] Killing /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-master with pid 5855 [ OK ] RaftConsensusNonVoterITest.TabletServerIsGoneAndBack (8499 ms) [ RUN ] RaftConsensusNonVoterITest.FailedTabletCopy I0507 06:31:39.876722 231 test_util.cc:212] Using random seed: 1587235154 I0507 06:31:39.878335 231 ts_itest-base.cc:112] Starting cluster with: I0507 06:31:39.878381 231 ts_itest-base.cc:113] -------------- I0507 06:31:39.878398 231 ts_itest-base.cc:114] 5 tablet servers I0507 06:31:39.878425 231 ts_itest-base.cc:115] 3 replicas per TS I0507 06:31:39.878482 231 ts_itest-base.cc:116] -------------- I0507 06:31:39.879683 231 external_mini_cluster.cc:933] Running /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-master /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-master --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.FailedTabletCopy.1557210500371863-231/raft_consensus-itest-cluster/master-0/wal --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.FailedTabletCopy.1557210500371863-231/raft_consensus-itest-cluster/master-0/data --block_manager=log --webserver_interface=localhost --ipki_ca_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.0.57.254:46549 --webserver_interface=127.0.57.254 --webserver_port=0 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.FailedTabletCopy.1557210500371863-231/raft_consensus-itest-cluster/master-0/logs --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.FailedTabletCopy.1557210500371863-231/raft_consensus-itest-cluster/master-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=true --tserver_unresponsive_timeout_ms=2500 --rpc_reuseport=true with env {} WARNING: Logging before InitGoogleLogging() is written to STDERR W0507 06:31:40.081620 6650 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0507 06:31:40.081857 6650 flags.cc:404] Enabled unsafe flag: --never_fsync=true W0507 06:31:40.084311 6650 flags.cc:404] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true W0507 06:31:40.084372 6650 flags.cc:404] Enabled experimental flag: --ipki_ca_key_size=1024 W0507 06:31:40.084406 6650 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W0507 06:31:40.084434 6650 flags.cc:404] Enabled experimental flag: --tsk_num_rsa_bits=512 W0507 06:31:40.084466 6650 flags.cc:404] Enabled experimental flag: --rpc_reuseport=true I0507 06:31:40.089061 6650 master_main.cc:73] Master server non-default flags: --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.FailedTabletCopy.1557210500371863-231/raft_consensus-itest-cluster/master-0/data --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.FailedTabletCopy.1557210500371863-231/raft_consensus-itest-cluster/master-0/wal --ipki_ca_key_size=1024 --tserver_unresponsive_timeout_ms=2500 --ipki_server_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.0.57.254:46549 --rpc_reuseport=true --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.FailedTabletCopy.1557210500371863-231/raft_consensus-itest-cluster/master-0/data/info.pb --webserver_interface=127.0.57.254 --webserver_port=0 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.FailedTabletCopy.1557210500371863-231/raft_consensus-itest-cluster/master-0/logs --logbuflevel=-1 --logtostderr=true Master server version: kudu 1.10.0-cdh6.x-SNAPSHOT revision 7d0ee26697926e87af9ef395008799577f6ab43f build type FASTDEBUG built by jenkins at 06 May 2019 22:44:21 PST on quasar-knmpfx-1.vpc.cloudera.com build id 322017 ASAN enabled I0507 06:31:40.095674 6650 system_ntp.cc:202] NTP initialized. Skew: 500ppm Current error: 124516us I0507 06:31:40.096995 6650 fs_manager.cc:263] Metadata directory not provided I0507 06:31:40.097074 6650 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0507 06:31:40.097371 6650 server_base.cc:433] Could not load existing FS layout: Not found: could not find a healthy instance file I0507 06:31:40.097434 6650 server_base.cc:434] Attempting to create new FS layout instead I0507 06:31:40.099968 6650 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.FailedTabletCopy.1557210500371863-231/raft_consensus-itest-cluster/master-0/data/instance: uuid: "1d33dc1d0f9b45e48cf062a0b2c31a67" format_stamp: "Formatted at 2019-05-07 06:31:40 on dist-test-slave-dist-test-slave-rqp6" I0507 06:31:40.100494 6650 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.FailedTabletCopy.1557210500371863-231/raft_consensus-itest-cluster/master-0/wal/instance: uuid: "1d33dc1d0f9b45e48cf062a0b2c31a67" format_stamp: "Formatted at 2019-05-07 06:31:40 on dist-test-slave-dist-test-slave-rqp6" I0507 06:31:40.107240 6650 fs_manager.cc:503] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s I0507 06:31:40.108306 6650 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0507 06:31:40.108949 6650 file_cache.cc:466] Constructed file cache lbm with capacity 419430 I0507 06:31:40.111981 6650 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s I0507 06:31:40.112054 6650 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.FailedTabletCopy.1557210500371863-231/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.FailedTabletCopy.1557210500371863-231/raft_consensus-itest-cluster/master-0/wal uuid: "1d33dc1d0f9b45e48cf062a0b2c31a67" format_stamp: "Formatted at 2019-05-07 06:31:40 on dist-test-slave-dist-test-slave-rqp6" I0507 06:31:40.112208 6650 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.FailedTabletCopy.1557210500371863-231/raft_consensus-itest-cluster/master-0/wal metadata directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.FailedTabletCopy.1557210500371863-231/raft_consensus-itest-cluster/master-0/wal 1 data directories: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.FailedTabletCopy.1557210500371863-231/raft_consensus-itest-cluster/master-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0507 06:31:40.144356 6650 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0507 06:31:40.159307 6650 rpc_server.cc:205] RPC server started. Bound to: 127.0.57.254:46549 I0507 06:31:40.160487 6650 webserver.cc:308] Webserver started at http://127.0.57.254:35685/ using document root and password file I0507 06:31:40.161238 6650 server_base.cc:608] Dumped server information to /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.FailedTabletCopy.1557210500371863-231/raft_consensus-itest-cluster/master-0/data/info.pb I0507 06:31:40.164590 6709 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0507 06:31:40.168684 231 external_mini_cluster.cc:995] Started /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-master as pid 6650 I0507 06:31:40.169737 231 external_mini_cluster.cc:933] Running /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.FailedTabletCopy.1557210500371863-231/raft_consensus-itest-cluster/ts-0/wal --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.FailedTabletCopy.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data --block_manager=log --rpc_bind_addresses=127.0.57.193:0 --local_ip_for_outbound_sockets=127.0.57.193 --webserver_interface=127.0.57.193 --webserver_port=0 --tserver_master_addrs=127.0.57.254:46549 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.FailedTabletCopy.1557210500371863-231/raft_consensus-itest-cluster/ts-0/logs --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.FailedTabletCopy.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=true --follower_unavailable_considered_failed_sec=10 --consensus_rpc_timeout_ms=5000 --log_compression_codec=none --log_cache_size_limit_mb=1 --log_segment_size_mb=1 --log_async_preallocate_segments=false --log_min_segments_to_retain=1 --log_max_segments_to_retain=3 --maintenance_manager_polling_interval_ms=100 --log_target_replay_size_mb=1 --max_cell_size_bytes=1000000 with env {} I0507 06:31:40.177866 6709 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 1d33dc1d0f9b45e48cf062a0b2c31a67: Bootstrap starting. I0507 06:31:40.181033 6709 tablet_bootstrap.cc:590] T 00000000000000000000000000000000 P 1d33dc1d0f9b45e48cf062a0b2c31a67: No blocks or log segments found. Creating new log. I0507 06:31:40.181975 6709 log.cc:525] T 00000000000000000000000000000000 P 1d33dc1d0f9b45e48cf062a0b2c31a67: Log is configured to *not* fsync() on all Append() calls I0507 06:31:40.190380 6709 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 1d33dc1d0f9b45e48cf062a0b2c31a67: No bootstrap required, opened a new log I0507 06:31:40.200821 6709 raft_consensus.cc:340] T 00000000000000000000000000000000 P 1d33dc1d0f9b45e48cf062a0b2c31a67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d33dc1d0f9b45e48cf062a0b2c31a67" member_type: VOTER } I0507 06:31:40.201232 6709 raft_consensus.cc:366] T 00000000000000000000000000000000 P 1d33dc1d0f9b45e48cf062a0b2c31a67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0507 06:31:40.201313 6709 raft_consensus.cc:705] T 00000000000000000000000000000000 P 1d33dc1d0f9b45e48cf062a0b2c31a67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d33dc1d0f9b45e48cf062a0b2c31a67, State: Initialized, Role: FOLLOWER I0507 06:31:40.201932 6709 consensus_queue.cc:229] T 00000000000000000000000000000000 P 1d33dc1d0f9b45e48cf062a0b2c31a67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d33dc1d0f9b45e48cf062a0b2c31a67" member_type: VOTER } I0507 06:31:40.202142 6709 raft_consensus.cc:380] T 00000000000000000000000000000000 P 1d33dc1d0f9b45e48cf062a0b2c31a67 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0507 06:31:40.202234 6709 raft_consensus.cc:472] T 00000000000000000000000000000000 P 1d33dc1d0f9b45e48cf062a0b2c31a67 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0507 06:31:40.202353 6709 raft_consensus.cc:2886] T 00000000000000000000000000000000 P 1d33dc1d0f9b45e48cf062a0b2c31a67 [term 0 FOLLOWER]: Advancing to term 1 I0507 06:31:40.205001 6709 raft_consensus.cc:494] T 00000000000000000000000000000000 P 1d33dc1d0f9b45e48cf062a0b2c31a67 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d33dc1d0f9b45e48cf062a0b2c31a67" member_type: VOTER } I0507 06:31:40.205657 6709 leader_election.cc:310] T 00000000000000000000000000000000 P 1d33dc1d0f9b45e48cf062a0b2c31a67 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1d33dc1d0f9b45e48cf062a0b2c31a67; no voters: I0507 06:31:40.208756 6709 leader_election.cc:296] T 00000000000000000000000000000000 P 1d33dc1d0f9b45e48cf062a0b2c31a67 [CANDIDATE]: Term 1 election: Requested vote from peers I0507 06:31:40.208950 6712 raft_consensus.cc:2640] T 00000000000000000000000000000000 P 1d33dc1d0f9b45e48cf062a0b2c31a67 [term 1 FOLLOWER]: Leader election won for term 1 I0507 06:31:40.209652 6712 raft_consensus.cc:667] T 00000000000000000000000000000000 P 1d33dc1d0f9b45e48cf062a0b2c31a67 [term 1 LEADER]: Becoming Leader. State: Replica: 1d33dc1d0f9b45e48cf062a0b2c31a67, State: Running, Role: LEADER I0507 06:31:40.210184 6712 consensus_queue.cc:206] T 00000000000000000000000000000000 P 1d33dc1d0f9b45e48cf062a0b2c31a67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d33dc1d0f9b45e48cf062a0b2c31a67" member_type: VOTER } I0507 06:31:40.211840 6712 sys_catalog.cc:337] T 00000000000000000000000000000000 P 1d33dc1d0f9b45e48cf062a0b2c31a67 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1d33dc1d0f9b45e48cf062a0b2c31a67. Latest consensus state: current_term: 1 leader_uuid: "1d33dc1d0f9b45e48cf062a0b2c31a67" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d33dc1d0f9b45e48cf062a0b2c31a67" member_type: VOTER } } I0507 06:31:40.212576 6712 sys_catalog.cc:340] T 00000000000000000000000000000000 P 1d33dc1d0f9b45e48cf062a0b2c31a67 [sys.catalog]: This master's current role is: LEADER I0507 06:31:40.212376 6714 sys_catalog.cc:337] T 00000000000000000000000000000000 P 1d33dc1d0f9b45e48cf062a0b2c31a67 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1d33dc1d0f9b45e48cf062a0b2c31a67" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d33dc1d0f9b45e48cf062a0b2c31a67" member_type: VOTER } } I0507 06:31:40.213114 6709 sys_catalog.cc:337] T 00000000000000000000000000000000 P 1d33dc1d0f9b45e48cf062a0b2c31a67 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 1 leader_uuid: "1d33dc1d0f9b45e48cf062a0b2c31a67" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d33dc1d0f9b45e48cf062a0b2c31a67" member_type: VOTER } } I0507 06:31:40.213870 6709 sys_catalog.cc:340] T 00000000000000000000000000000000 P 1d33dc1d0f9b45e48cf062a0b2c31a67 [sys.catalog]: This master's current role is: LEADER I0507 06:31:40.213872 6714 sys_catalog.cc:340] T 00000000000000000000000000000000 P 1d33dc1d0f9b45e48cf062a0b2c31a67 [sys.catalog]: This master's current role is: LEADER I0507 06:31:40.215791 6709 sys_catalog.cc:424] T 00000000000000000000000000000000 P 1d33dc1d0f9b45e48cf062a0b2c31a67 [sys.catalog]: configured and running, proceeding with master startup. I0507 06:31:40.215873 6716 catalog_manager.cc:1055] Loading table and tablet metadata into memory... I0507 06:31:40.216550 6718 catalog_manager.cc:538] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting I0507 06:31:40.222918 6716 catalog_manager.cc:1066] Initializing Kudu internal certificate authority... I0507 06:31:40.270539 6716 catalog_manager.cc:950] Generated new certificate authority record I0507 06:31:40.271850 6716 catalog_manager.cc:1075] Loading token signing keys... I0507 06:31:40.286641 6716 catalog_manager.cc:4292] T 00000000000000000000000000000000 P 1d33dc1d0f9b45e48cf062a0b2c31a67: Generated new TSK 0 WARNING: Logging before InitGoogleLogging() is written to STDERR W0507 06:31:40.344482 6710 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0507 06:31:40.344785 6710 flags.cc:404] Enabled unsafe flag: --max_cell_size_bytes=1000000 W0507 06:31:40.344848 6710 flags.cc:404] Enabled unsafe flag: --never_fsync=true W0507 06:31:40.346994 6710 flags.cc:404] Enabled experimental flag: --log_compression_codec=none W0507 06:31:40.347055 6710 flags.cc:404] Enabled experimental flag: --log_max_segments_to_retain=3 W0507 06:31:40.347090 6710 flags.cc:404] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true W0507 06:31:40.347147 6710 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W0507 06:31:40.347227 6710 flags.cc:404] Enabled experimental flag: --log_target_replay_size_mb=1 W0507 06:31:40.347283 6710 flags.cc:404] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.57.193 I0507 06:31:40.351490 6710 tablet_server_main.cc:78] Tablet server non-default flags: --consensus_rpc_timeout_ms=5000 --follower_unavailable_considered_failed_sec=10 --log_compression_codec=none --log_max_segments_to_retain=3 --log_cache_size_limit_mb=1 --log_async_preallocate_segments=false --log_segment_size_mb=1 --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.FailedTabletCopy.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.FailedTabletCopy.1557210500371863-231/raft_consensus-itest-cluster/ts-0/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.0.57.193:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.FailedTabletCopy.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data/info.pb --webserver_interface=127.0.57.193 --webserver_port=0 --max_cell_size_bytes=1000000 --tserver_master_addrs=127.0.57.254:46549 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --log_target_replay_size_mb=1 --maintenance_manager_polling_interval_ms=100 --enable_minidumps=false --local_ip_for_outbound_sockets=127.0.57.193 --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.FailedTabletCopy.1557210500371863-231/raft_consensus-itest-cluster/ts-0/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.10.0-cdh6.x-SNAPSHOT revision 7d0ee26697926e87af9ef395008799577f6ab43f build type FASTDEBUG built by jenkins at 06 May 2019 22:44:21 PST on quasar-knmpfx-1.vpc.cloudera.com build id 322017 ASAN enabled I0507 06:31:40.358933 6710 system_ntp.cc:202] NTP initialized. Skew: 500ppm Current error: 124516us I0507 06:31:40.359575 6710 fs_manager.cc:263] Metadata directory not provided I0507 06:31:40.359635 6710 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0507 06:31:40.359907 6710 server_base.cc:433] Could not load existing FS layout: Not found: could not find a healthy instance file I0507 06:31:40.360007 6710 server_base.cc:434] Attempting to create new FS layout instead I0507 06:31:40.362423 6710 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.FailedTabletCopy.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data/instance: uuid: "18b5d1299d9146ccabb7d4819ec95db1" format_stamp: "Formatted at 2019-05-07 06:31:40 on dist-test-slave-dist-test-slave-rqp6" I0507 06:31:40.363027 6710 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.FailedTabletCopy.1557210500371863-231/raft_consensus-itest-cluster/ts-0/wal/instance: uuid: "18b5d1299d9146ccabb7d4819ec95db1" format_stamp: "Formatted at 2019-05-07 06:31:40 on dist-test-slave-dist-test-slave-rqp6" I0507 06:31:40.369268 6710 fs_manager.cc:503] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s I0507 06:31:40.370337 6710 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0507 06:31:40.370884 6710 file_cache.cc:466] Constructed file cache lbm with capacity 419430 I0507 06:31:40.373672 6710 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s I0507 06:31:40.373786 6710 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.FailedTabletCopy.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.FailedTabletCopy.1557210500371863-231/raft_consensus-itest-cluster/ts-0/wal uuid: "18b5d1299d9146ccabb7d4819ec95db1" format_stamp: "Formatted at 2019-05-07 06:31:40 on dist-test-slave-dist-test-slave-rqp6" I0507 06:31:40.373940 6710 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.FailedTabletCopy.1557210500371863-231/raft_consensus-itest-cluster/ts-0/wal metadata directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.FailedTabletCopy.1557210500371863-231/raft_consensus-itest-cluster/ts-0/wal 1 data directories: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.FailedTabletCopy.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0507 06:31:40.429011 6710 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0507 06:31:40.430709 6710 ts_tablet_manager.cc:356] Loaded tablet metadata (0 total tablets, 0 live tablets) I0507 06:31:40.430783 6710 ts_tablet_manager.cc:376] Registered 0 tablets I0507 06:31:40.463464 6710 rpc_server.cc:205] RPC server started. Bound to: 127.0.57.193:35039 I0507 06:31:40.464566 6710 webserver.cc:308] Webserver started at http://127.0.57.193:44829/ using document root and password file I0507 06:31:40.465209 6710 server_base.cc:608] Dumped server information to /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.FailedTabletCopy.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data/info.pb I0507 06:31:40.475311 231 external_mini_cluster.cc:995] Started /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver as pid 6710 I0507 06:31:40.475970 231 external_mini_cluster.cc:933] Running /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.FailedTabletCopy.1557210500371863-231/raft_consensus-itest-cluster/ts-1/wal --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.FailedTabletCopy.1557210500371863-231/raft_consensus-itest-cluster/ts-1/data --block_manager=log --rpc_bind_addresses=127.0.57.194:0 --local_ip_for_outbound_sockets=127.0.57.194 --webserver_interface=127.0.57.194 --webserver_port=0 --tserver_master_addrs=127.0.57.254:46549 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.FailedTabletCopy.1557210500371863-231/raft_consensus-itest-cluster/ts-1/logs --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.FailedTabletCopy.1557210500371863-231/raft_consensus-itest-cluster/ts-1/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=true --follower_unavailable_considered_failed_sec=10 --consensus_rpc_timeout_ms=5000 --log_compression_codec=none --log_cache_size_limit_mb=1 --log_segment_size_mb=1 --log_async_preallocate_segments=false --log_min_segments_to_retain=1 --log_max_segments_to_retain=3 --maintenance_manager_polling_interval_ms=100 --log_target_replay_size_mb=1 --max_cell_size_bytes=1000000 with env {} I0507 06:31:40.489929 6839 heartbeater.cc:345] Connected to a master server at 127.0.57.254:46549 I0507 06:31:40.490128 6839 heartbeater.cc:426] Registering TS with master... I0507 06:31:40.490629 6839 heartbeater.cc:475] Master 127.0.57.254:46549 requested a full tablet report, sending... I0507 06:31:40.492296 6676 ts_manager.cc:159] Registered new tserver with Master: 18b5d1299d9146ccabb7d4819ec95db1 (127.0.57.193:35039) I0507 06:31:40.494549 6676 master_service.cc:273] Signed X509 certificate for tserver {username='slave'} at 127.0.57.193:44211 WARNING: Logging before InitGoogleLogging() is written to STDERR W0507 06:31:40.646172 6843 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0507 06:31:40.646433 6843 flags.cc:404] Enabled unsafe flag: --max_cell_size_bytes=1000000 W0507 06:31:40.646481 6843 flags.cc:404] Enabled unsafe flag: --never_fsync=true W0507 06:31:40.648643 6843 flags.cc:404] Enabled experimental flag: --log_compression_codec=none W0507 06:31:40.648715 6843 flags.cc:404] Enabled experimental flag: --log_max_segments_to_retain=3 W0507 06:31:40.648756 6843 flags.cc:404] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true W0507 06:31:40.648806 6843 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W0507 06:31:40.648905 6843 flags.cc:404] Enabled experimental flag: --log_target_replay_size_mb=1 W0507 06:31:40.648948 6843 flags.cc:404] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.57.194 I0507 06:31:40.653180 6843 tablet_server_main.cc:78] Tablet server non-default flags: --consensus_rpc_timeout_ms=5000 --follower_unavailable_considered_failed_sec=10 --log_compression_codec=none --log_max_segments_to_retain=3 --log_cache_size_limit_mb=1 --log_async_preallocate_segments=false --log_segment_size_mb=1 --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.FailedTabletCopy.1557210500371863-231/raft_consensus-itest-cluster/ts-1/data --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.FailedTabletCopy.1557210500371863-231/raft_consensus-itest-cluster/ts-1/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.0.57.194:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.FailedTabletCopy.1557210500371863-231/raft_consensus-itest-cluster/ts-1/data/info.pb --webserver_interface=127.0.57.194 --webserver_port=0 --max_cell_size_bytes=1000000 --tserver_master_addrs=127.0.57.254:46549 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --log_target_replay_size_mb=1 --maintenance_manager_polling_interval_ms=100 --enable_minidumps=false --local_ip_for_outbound_sockets=127.0.57.194 --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.FailedTabletCopy.1557210500371863-231/raft_consensus-itest-cluster/ts-1/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.10.0-cdh6.x-SNAPSHOT revision 7d0ee26697926e87af9ef395008799577f6ab43f build type FASTDEBUG built by jenkins at 06 May 2019 22:44:21 PST on quasar-knmpfx-1.vpc.cloudera.com build id 322017 ASAN enabled I0507 06:31:40.661824 6843 system_ntp.cc:202] NTP initialized. Skew: 500ppm Current error: 124516us I0507 06:31:40.662578 6843 fs_manager.cc:263] Metadata directory not provided I0507 06:31:40.662662 6843 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0507 06:31:40.662963 6843 server_base.cc:433] Could not load existing FS layout: Not found: could not find a healthy instance file I0507 06:31:40.663033 6843 server_base.cc:434] Attempting to create new FS layout instead I0507 06:31:40.665724 6843 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.FailedTabletCopy.1557210500371863-231/raft_consensus-itest-cluster/ts-1/data/instance: uuid: "e84d1eaf01124e6dbc9be1f1fde546b7" format_stamp: "Formatted at 2019-05-07 06:31:40 on dist-test-slave-dist-test-slave-rqp6" I0507 06:31:40.666311 6843 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.FailedTabletCopy.1557210500371863-231/raft_consensus-itest-cluster/ts-1/wal/instance: uuid: "e84d1eaf01124e6dbc9be1f1fde546b7" format_stamp: "Formatted at 2019-05-07 06:31:40 on dist-test-slave-dist-test-slave-rqp6" I0507 06:31:40.673521 6843 fs_manager.cc:503] Time spent creating directory manager: real 0.007s user 0.002s sys 0.005s I0507 06:31:40.674643 6843 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0507 06:31:40.675307 6843 file_cache.cc:466] Constructed file cache lbm with capacity 419430 I0507 06:31:40.679198 6843 fs_manager.cc:419] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s I0507 06:31:40.679294 6843 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.FailedTabletCopy.1557210500371863-231/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.FailedTabletCopy.1557210500371863-231/raft_consensus-itest-cluster/ts-1/wal uuid: "e84d1eaf01124e6dbc9be1f1fde546b7" format_stamp: "Formatted at 2019-05-07 06:31:40 on dist-test-slave-dist-test-slave-rqp6" I0507 06:31:40.679483 6843 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.FailedTabletCopy.1557210500371863-231/raft_consensus-itest-cluster/ts-1/wal metadata directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.FailedTabletCopy.1557210500371863-231/raft_consensus-itest-cluster/ts-1/wal 1 data directories: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.FailedTabletCopy.1557210500371863-231/raft_consensus-itest-cluster/ts-1/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0507 06:31:40.726598 6843 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0507 06:31:40.728509 6843 ts_tablet_manager.cc:356] Loaded tablet metadata (0 total tablets, 0 live tablets) I0507 06:31:40.728636 6843 ts_tablet_manager.cc:376] Registered 0 tablets I0507 06:31:40.757722 6843 rpc_server.cc:205] RPC server started. Bound to: 127.0.57.194:37459 I0507 06:31:40.759069 6843 webserver.cc:308] Webserver started at http://127.0.57.194:33367/ using document root and password file I0507 06:31:40.759876 6843 server_base.cc:608] Dumped server information to /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.FailedTabletCopy.1557210500371863-231/raft_consensus-itest-cluster/ts-1/data/info.pb I0507 06:31:40.766170 231 external_mini_cluster.cc:995] Started /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver as pid 6843 I0507 06:31:40.767058 231 external_mini_cluster.cc:933] Running /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.FailedTabletCopy.1557210500371863-231/raft_consensus-itest-cluster/ts-2/wal --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.FailedTabletCopy.1557210500371863-231/raft_consensus-itest-cluster/ts-2/data --block_manager=log --rpc_bind_addresses=127.0.57.195:0 --local_ip_for_outbound_sockets=127.0.57.195 --webserver_interface=127.0.57.195 --webserver_port=0 --tserver_master_addrs=127.0.57.254:46549 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.FailedTabletCopy.1557210500371863-231/raft_consensus-itest-cluster/ts-2/logs --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.FailedTabletCopy.1557210500371863-231/raft_consensus-itest-cluster/ts-2/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=true --follower_unavailable_considered_failed_sec=10 --consensus_rpc_timeout_ms=5000 --log_compression_codec=none --log_cache_size_limit_mb=1 --log_segment_size_mb=1 --log_async_preallocate_segments=false --log_min_segments_to_retain=1 --log_max_segments_to_retain=3 --maintenance_manager_polling_interval_ms=100 --log_target_replay_size_mb=1 --max_cell_size_bytes=1000000 with env {} I0507 06:31:40.772872 6962 heartbeater.cc:345] Connected to a master server at 127.0.57.254:46549 I0507 06:31:40.773083 6962 heartbeater.cc:426] Registering TS with master... I0507 06:31:40.773579 6962 heartbeater.cc:475] Master 127.0.57.254:46549 requested a full tablet report, sending... I0507 06:31:40.774860 6676 ts_manager.cc:159] Registered new tserver with Master: e84d1eaf01124e6dbc9be1f1fde546b7 (127.0.57.194:37459) I0507 06:31:40.775815 6676 master_service.cc:273] Signed X509 certificate for tserver {username='slave'} at 127.0.57.194:49689 WARNING: Logging before InitGoogleLogging() is written to STDERR W0507 06:31:40.928225 6965 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0507 06:31:40.928452 6965 flags.cc:404] Enabled unsafe flag: --max_cell_size_bytes=1000000 W0507 06:31:40.928495 6965 flags.cc:404] Enabled unsafe flag: --never_fsync=true W0507 06:31:40.930557 6965 flags.cc:404] Enabled experimental flag: --log_compression_codec=none W0507 06:31:40.930619 6965 flags.cc:404] Enabled experimental flag: --log_max_segments_to_retain=3 W0507 06:31:40.930662 6965 flags.cc:404] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true W0507 06:31:40.930716 6965 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W0507 06:31:40.930811 6965 flags.cc:404] Enabled experimental flag: --log_target_replay_size_mb=1 W0507 06:31:40.930866 6965 flags.cc:404] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.57.195 I0507 06:31:40.934943 6965 tablet_server_main.cc:78] Tablet server non-default flags: --consensus_rpc_timeout_ms=5000 --follower_unavailable_considered_failed_sec=10 --log_compression_codec=none --log_max_segments_to_retain=3 --log_cache_size_limit_mb=1 --log_async_preallocate_segments=false --log_segment_size_mb=1 --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.FailedTabletCopy.1557210500371863-231/raft_consensus-itest-cluster/ts-2/data --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.FailedTabletCopy.1557210500371863-231/raft_consensus-itest-cluster/ts-2/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.0.57.195:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.FailedTabletCopy.1557210500371863-231/raft_consensus-itest-cluster/ts-2/data/info.pb --webserver_interface=127.0.57.195 --webserver_port=0 --max_cell_size_bytes=1000000 --tserver_master_addrs=127.0.57.254:46549 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --log_target_replay_size_mb=1 --maintenance_manager_polling_interval_ms=100 --enable_minidumps=false --local_ip_for_outbound_sockets=127.0.57.195 --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.FailedTabletCopy.1557210500371863-231/raft_consensus-itest-cluster/ts-2/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.10.0-cdh6.x-SNAPSHOT revision 7d0ee26697926e87af9ef395008799577f6ab43f build type FASTDEBUG built by jenkins at 06 May 2019 22:44:21 PST on quasar-knmpfx-1.vpc.cloudera.com build id 322017 ASAN enabled I0507 06:31:40.942229 6965 system_ntp.cc:202] NTP initialized. Skew: 500ppm Current error: 124516us I0507 06:31:40.942889 6965 fs_manager.cc:263] Metadata directory not provided I0507 06:31:40.942953 6965 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0507 06:31:40.943176 6965 server_base.cc:433] Could not load existing FS layout: Not found: could not find a healthy instance file I0507 06:31:40.943238 6965 server_base.cc:434] Attempting to create new FS layout instead I0507 06:31:40.945988 6965 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.FailedTabletCopy.1557210500371863-231/raft_consensus-itest-cluster/ts-2/data/instance: uuid: "e2ec75f7edc34d4aa7aca3a4843cf259" format_stamp: "Formatted at 2019-05-07 06:31:40 on dist-test-slave-dist-test-slave-rqp6" I0507 06:31:40.946518 6965 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.FailedTabletCopy.1557210500371863-231/raft_consensus-itest-cluster/ts-2/wal/instance: uuid: "e2ec75f7edc34d4aa7aca3a4843cf259" format_stamp: "Formatted at 2019-05-07 06:31:40 on dist-test-slave-dist-test-slave-rqp6" I0507 06:31:40.953184 6965 fs_manager.cc:503] Time spent creating directory manager: real 0.006s user 0.002s sys 0.005s I0507 06:31:40.954262 6965 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0507 06:31:40.954874 6965 file_cache.cc:466] Constructed file cache lbm with capacity 419430 I0507 06:31:40.958161 6965 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s I0507 06:31:40.958256 6965 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.FailedTabletCopy.1557210500371863-231/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.FailedTabletCopy.1557210500371863-231/raft_consensus-itest-cluster/ts-2/wal uuid: "e2ec75f7edc34d4aa7aca3a4843cf259" format_stamp: "Formatted at 2019-05-07 06:31:40 on dist-test-slave-dist-test-slave-rqp6" I0507 06:31:40.958413 6965 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.FailedTabletCopy.1557210500371863-231/raft_consensus-itest-cluster/ts-2/wal metadata directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.FailedTabletCopy.1557210500371863-231/raft_consensus-itest-cluster/ts-2/wal 1 data directories: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.FailedTabletCopy.1557210500371863-231/raft_consensus-itest-cluster/ts-2/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0507 06:31:40.982237 6965 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0507 06:31:40.983891 6965 ts_tablet_manager.cc:356] Loaded tablet metadata (0 total tablets, 0 live tablets) I0507 06:31:40.983968 6965 ts_tablet_manager.cc:376] Registered 0 tablets I0507 06:31:41.011229 6965 rpc_server.cc:205] RPC server started. Bound to: 127.0.57.195:35003 I0507 06:31:41.012411 6965 webserver.cc:308] Webserver started at http://127.0.57.195:42757/ using document root and password file I0507 06:31:41.013057 6965 server_base.cc:608] Dumped server information to /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.FailedTabletCopy.1557210500371863-231/raft_consensus-itest-cluster/ts-2/data/info.pb I0507 06:31:41.015779 231 external_mini_cluster.cc:995] Started /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver as pid 6965 I0507 06:31:41.016510 231 external_mini_cluster.cc:933] Running /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.FailedTabletCopy.1557210500371863-231/raft_consensus-itest-cluster/ts-3/wal --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.FailedTabletCopy.1557210500371863-231/raft_consensus-itest-cluster/ts-3/data --block_manager=log --rpc_bind_addresses=127.0.57.196:0 --local_ip_for_outbound_sockets=127.0.57.196 --webserver_interface=127.0.57.196 --webserver_port=0 --tserver_master_addrs=127.0.57.254:46549 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.FailedTabletCopy.1557210500371863-231/raft_consensus-itest-cluster/ts-3/logs --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.FailedTabletCopy.1557210500371863-231/raft_consensus-itest-cluster/ts-3/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=true --follower_unavailable_considered_failed_sec=10 --consensus_rpc_timeout_ms=5000 --log_compression_codec=none --log_cache_size_limit_mb=1 --log_segment_size_mb=1 --log_async_preallocate_segments=false --log_min_segments_to_retain=1 --log_max_segments_to_retain=3 --maintenance_manager_polling_interval_ms=100 --log_target_replay_size_mb=1 --max_cell_size_bytes=1000000 with env {} I0507 06:31:41.026829 7084 heartbeater.cc:345] Connected to a master server at 127.0.57.254:46549 I0507 06:31:41.027071 7084 heartbeater.cc:426] Registering TS with master... I0507 06:31:41.027678 7084 heartbeater.cc:475] Master 127.0.57.254:46549 requested a full tablet report, sending... I0507 06:31:41.028928 6676 ts_manager.cc:159] Registered new tserver with Master: e2ec75f7edc34d4aa7aca3a4843cf259 (127.0.57.195:35003) I0507 06:31:41.029923 6676 master_service.cc:273] Signed X509 certificate for tserver {username='slave'} at 127.0.57.195:53139 WARNING: Logging before InitGoogleLogging() is written to STDERR W0507 06:31:41.179785 7087 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0507 06:31:41.179988 7087 flags.cc:404] Enabled unsafe flag: --max_cell_size_bytes=1000000 W0507 06:31:41.180045 7087 flags.cc:404] Enabled unsafe flag: --never_fsync=true W0507 06:31:41.182250 7087 flags.cc:404] Enabled experimental flag: --log_compression_codec=none W0507 06:31:41.182308 7087 flags.cc:404] Enabled experimental flag: --log_max_segments_to_retain=3 W0507 06:31:41.182341 7087 flags.cc:404] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true W0507 06:31:41.182386 7087 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W0507 06:31:41.182459 7087 flags.cc:404] Enabled experimental flag: --log_target_replay_size_mb=1 W0507 06:31:41.182504 7087 flags.cc:404] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.57.196 I0507 06:31:41.186627 7087 tablet_server_main.cc:78] Tablet server non-default flags: --consensus_rpc_timeout_ms=5000 --follower_unavailable_considered_failed_sec=10 --log_compression_codec=none --log_max_segments_to_retain=3 --log_cache_size_limit_mb=1 --log_async_preallocate_segments=false --log_segment_size_mb=1 --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.FailedTabletCopy.1557210500371863-231/raft_consensus-itest-cluster/ts-3/data --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.FailedTabletCopy.1557210500371863-231/raft_consensus-itest-cluster/ts-3/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.0.57.196:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.FailedTabletCopy.1557210500371863-231/raft_consensus-itest-cluster/ts-3/data/info.pb --webserver_interface=127.0.57.196 --webserver_port=0 --max_cell_size_bytes=1000000 --tserver_master_addrs=127.0.57.254:46549 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --log_target_replay_size_mb=1 --maintenance_manager_polling_interval_ms=100 --enable_minidumps=false --local_ip_for_outbound_sockets=127.0.57.196 --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.FailedTabletCopy.1557210500371863-231/raft_consensus-itest-cluster/ts-3/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.10.0-cdh6.x-SNAPSHOT revision 7d0ee26697926e87af9ef395008799577f6ab43f build type FASTDEBUG built by jenkins at 06 May 2019 22:44:21 PST on quasar-knmpfx-1.vpc.cloudera.com build id 322017 ASAN enabled I0507 06:31:41.194591 7087 system_ntp.cc:202] NTP initialized. Skew: 500ppm Current error: 125016us I0507 06:31:41.195247 7087 fs_manager.cc:263] Metadata directory not provided I0507 06:31:41.195309 7087 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0507 06:31:41.195652 7087 server_base.cc:433] Could not load existing FS layout: Not found: could not find a healthy instance file I0507 06:31:41.195721 7087 server_base.cc:434] Attempting to create new FS layout instead I0507 06:31:41.198112 7087 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.FailedTabletCopy.1557210500371863-231/raft_consensus-itest-cluster/ts-3/data/instance: uuid: "8c31c26018bd4f6391b7f63f1af2a6a7" format_stamp: "Formatted at 2019-05-07 06:31:41 on dist-test-slave-dist-test-slave-rqp6" I0507 06:31:41.198777 7087 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.FailedTabletCopy.1557210500371863-231/raft_consensus-itest-cluster/ts-3/wal/instance: uuid: "8c31c26018bd4f6391b7f63f1af2a6a7" format_stamp: "Formatted at 2019-05-07 06:31:41 on dist-test-slave-dist-test-slave-rqp6" I0507 06:31:41.205624 7087 fs_manager.cc:503] Time spent creating directory manager: real 0.007s user 0.002s sys 0.005s I0507 06:31:41.206759 7087 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0507 06:31:41.207312 7087 file_cache.cc:466] Constructed file cache lbm with capacity 419430 I0507 06:31:41.210088 7087 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s I0507 06:31:41.210161 7087 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.FailedTabletCopy.1557210500371863-231/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.FailedTabletCopy.1557210500371863-231/raft_consensus-itest-cluster/ts-3/wal uuid: "8c31c26018bd4f6391b7f63f1af2a6a7" format_stamp: "Formatted at 2019-05-07 06:31:41 on dist-test-slave-dist-test-slave-rqp6" I0507 06:31:41.210294 7087 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.FailedTabletCopy.1557210500371863-231/raft_consensus-itest-cluster/ts-3/wal metadata directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.FailedTabletCopy.1557210500371863-231/raft_consensus-itest-cluster/ts-3/wal 1 data directories: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.FailedTabletCopy.1557210500371863-231/raft_consensus-itest-cluster/ts-3/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0507 06:31:41.265535 7087 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0507 06:31:41.267505 7087 ts_tablet_manager.cc:356] Loaded tablet metadata (0 total tablets, 0 live tablets) I0507 06:31:41.267597 7087 ts_tablet_manager.cc:376] Registered 0 tablets I0507 06:31:41.302156 7087 rpc_server.cc:205] RPC server started. Bound to: 127.0.57.196:37823 I0507 06:31:41.303315 7087 webserver.cc:308] Webserver started at http://127.0.57.196:43293/ using document root and password file I0507 06:31:41.303969 7087 server_base.cc:608] Dumped server information to /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.FailedTabletCopy.1557210500371863-231/raft_consensus-itest-cluster/ts-3/data/info.pb I0507 06:31:41.306135 231 external_mini_cluster.cc:995] Started /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver as pid 7087 I0507 06:31:41.306744 231 external_mini_cluster.cc:933] Running /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.FailedTabletCopy.1557210500371863-231/raft_consensus-itest-cluster/ts-4/wal --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.FailedTabletCopy.1557210500371863-231/raft_consensus-itest-cluster/ts-4/data --block_manager=log --rpc_bind_addresses=127.0.57.197:0 --local_ip_for_outbound_sockets=127.0.57.197 --webserver_interface=127.0.57.197 --webserver_port=0 --tserver_master_addrs=127.0.57.254:46549 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.FailedTabletCopy.1557210500371863-231/raft_consensus-itest-cluster/ts-4/logs --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.FailedTabletCopy.1557210500371863-231/raft_consensus-itest-cluster/ts-4/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=true --follower_unavailable_considered_failed_sec=10 --consensus_rpc_timeout_ms=5000 --log_compression_codec=none --log_cache_size_limit_mb=1 --log_segment_size_mb=1 --log_async_preallocate_segments=false --log_min_segments_to_retain=1 --log_max_segments_to_retain=3 --maintenance_manager_polling_interval_ms=100 --log_target_replay_size_mb=1 --max_cell_size_bytes=1000000 with env {} I0507 06:31:41.322226 7206 heartbeater.cc:345] Connected to a master server at 127.0.57.254:46549 I0507 06:31:41.322438 7206 heartbeater.cc:426] Registering TS with master... I0507 06:31:41.322957 7206 heartbeater.cc:475] Master 127.0.57.254:46549 requested a full tablet report, sending... I0507 06:31:41.324095 6676 ts_manager.cc:159] Registered new tserver with Master: 8c31c26018bd4f6391b7f63f1af2a6a7 (127.0.57.196:37823) I0507 06:31:41.325059 6676 master_service.cc:273] Signed X509 certificate for tserver {username='slave'} at 127.0.57.196:52961 WARNING: Logging before InitGoogleLogging() is written to STDERR W0507 06:31:41.465289 7209 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0507 06:31:41.465564 7209 flags.cc:404] Enabled unsafe flag: --max_cell_size_bytes=1000000 W0507 06:31:41.465644 7209 flags.cc:404] Enabled unsafe flag: --never_fsync=true W0507 06:31:41.467815 7209 flags.cc:404] Enabled experimental flag: --log_compression_codec=none W0507 06:31:41.467881 7209 flags.cc:404] Enabled experimental flag: --log_max_segments_to_retain=3 W0507 06:31:41.467931 7209 flags.cc:404] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true W0507 06:31:41.467994 7209 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W0507 06:31:41.468111 7209 flags.cc:404] Enabled experimental flag: --log_target_replay_size_mb=1 W0507 06:31:41.468173 7209 flags.cc:404] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.57.197 I0507 06:31:41.472353 7209 tablet_server_main.cc:78] Tablet server non-default flags: --consensus_rpc_timeout_ms=5000 --follower_unavailable_considered_failed_sec=10 --log_compression_codec=none --log_max_segments_to_retain=3 --log_cache_size_limit_mb=1 --log_async_preallocate_segments=false --log_segment_size_mb=1 --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.FailedTabletCopy.1557210500371863-231/raft_consensus-itest-cluster/ts-4/data --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.FailedTabletCopy.1557210500371863-231/raft_consensus-itest-cluster/ts-4/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.0.57.197:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.FailedTabletCopy.1557210500371863-231/raft_consensus-itest-cluster/ts-4/data/info.pb --webserver_interface=127.0.57.197 --webserver_port=0 --max_cell_size_bytes=1000000 --tserver_master_addrs=127.0.57.254:46549 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --log_target_replay_size_mb=1 --maintenance_manager_polling_interval_ms=100 --enable_minidumps=false --local_ip_for_outbound_sockets=127.0.57.197 --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.FailedTabletCopy.1557210500371863-231/raft_consensus-itest-cluster/ts-4/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.10.0-cdh6.x-SNAPSHOT revision 7d0ee26697926e87af9ef395008799577f6ab43f build type FASTDEBUG built by jenkins at 06 May 2019 22:44:21 PST on quasar-knmpfx-1.vpc.cloudera.com build id 322017 ASAN enabled I0507 06:31:41.480974 7209 system_ntp.cc:202] NTP initialized. Skew: 500ppm Current error: 125016us I0507 06:31:41.481743 7209 fs_manager.cc:263] Metadata directory not provided I0507 06:31:41.481815 7209 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0507 06:31:41.482121 7209 server_base.cc:433] Could not load existing FS layout: Not found: could not find a healthy instance file I0507 06:31:41.482189 7209 server_base.cc:434] Attempting to create new FS layout instead I0507 06:31:41.485019 7209 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.FailedTabletCopy.1557210500371863-231/raft_consensus-itest-cluster/ts-4/data/instance: uuid: "48cc028a4cfd4637a8ede1b4e421ae8c" format_stamp: "Formatted at 2019-05-07 06:31:41 on dist-test-slave-dist-test-slave-rqp6" I0507 06:31:41.485633 7209 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.FailedTabletCopy.1557210500371863-231/raft_consensus-itest-cluster/ts-4/wal/instance: uuid: "48cc028a4cfd4637a8ede1b4e421ae8c" format_stamp: "Formatted at 2019-05-07 06:31:41 on dist-test-slave-dist-test-slave-rqp6" I0507 06:31:41.492394 7209 fs_manager.cc:503] Time spent creating directory manager: real 0.007s user 0.006s sys 0.000s I0507 06:31:41.493566 7209 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0507 06:31:41.494208 7209 file_cache.cc:466] Constructed file cache lbm with capacity 419430 I0507 06:31:41.497318 7209 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s I0507 06:31:41.497400 7209 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.FailedTabletCopy.1557210500371863-231/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.FailedTabletCopy.1557210500371863-231/raft_consensus-itest-cluster/ts-4/wal uuid: "48cc028a4cfd4637a8ede1b4e421ae8c" format_stamp: "Formatted at 2019-05-07 06:31:41 on dist-test-slave-dist-test-slave-rqp6" I0507 06:31:41.497565 7209 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.FailedTabletCopy.1557210500371863-231/raft_consensus-itest-cluster/ts-4/wal metadata directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.FailedTabletCopy.1557210500371863-231/raft_consensus-itest-cluster/ts-4/wal 1 data directories: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.FailedTabletCopy.1557210500371863-231/raft_consensus-itest-cluster/ts-4/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0507 06:31:41.498816 6839 heartbeater.cc:467] Master 127.0.57.254:46549 was elected leader, sending a full tablet report... I0507 06:31:41.514761 7209 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0507 06:31:41.516546 7209 ts_tablet_manager.cc:356] Loaded tablet metadata (0 total tablets, 0 live tablets) I0507 06:31:41.516674 7209 ts_tablet_manager.cc:376] Registered 0 tablets I0507 06:31:41.543928 7209 rpc_server.cc:205] RPC server started. Bound to: 127.0.57.197:40311 I0507 06:31:41.545203 7209 webserver.cc:308] Webserver started at http://127.0.57.197:39461/ using document root and password file I0507 06:31:41.545897 7209 server_base.cc:608] Dumped server information to /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.FailedTabletCopy.1557210500371863-231/raft_consensus-itest-cluster/ts-4/data/info.pb I0507 06:31:41.555203 231 external_mini_cluster.cc:995] Started /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver as pid 7209 I0507 06:31:41.559865 7328 heartbeater.cc:345] Connected to a master server at 127.0.57.254:46549 I0507 06:31:41.560122 7328 heartbeater.cc:426] Registering TS with master... I0507 06:31:41.560770 7328 heartbeater.cc:475] Master 127.0.57.254:46549 requested a full tablet report, sending... I0507 06:31:41.562047 6676 ts_manager.cc:159] Registered new tserver with Master: 48cc028a4cfd4637a8ede1b4e421ae8c (127.0.57.197:40311) I0507 06:31:41.563217 6676 master_service.cc:273] Signed X509 certificate for tserver {username='slave'} at 127.0.57.197:40769 I0507 06:31:41.567262 231 external_mini_cluster.cc:623] 5 TS(s) registered with all masters I0507 06:31:41.578795 6676 catalog_manager.cc:1415] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59590: name: "TestTable" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 3 split_rows_range_bounds { } partition_schema { range_schema { columns { name: "key" } } } I0507 06:31:41.610435 7143 tablet_service.cc:1016] Processing CreateTablet for tablet 558f015eda7b4593b2bf9090cc253899 (table=TestTable [id=d11606250418492493f264f029452e41]), partition=RANGE (key) PARTITION UNBOUNDED I0507 06:31:41.611748 7143 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet 558f015eda7b4593b2bf9090cc253899. 1 dirs total, 0 dirs full, 0 dirs failed I0507 06:31:41.613466 7021 tablet_service.cc:1016] Processing CreateTablet for tablet 558f015eda7b4593b2bf9090cc253899 (table=TestTable [id=d11606250418492493f264f029452e41]), partition=RANGE (key) PARTITION UNBOUNDED I0507 06:31:41.614334 7021 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet 558f015eda7b4593b2bf9090cc253899. 1 dirs total, 0 dirs full, 0 dirs failed I0507 06:31:41.618875 7265 tablet_service.cc:1016] Processing CreateTablet for tablet 558f015eda7b4593b2bf9090cc253899 (table=TestTable [id=d11606250418492493f264f029452e41]), partition=RANGE (key) PARTITION UNBOUNDED I0507 06:31:41.619915 7265 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet 558f015eda7b4593b2bf9090cc253899. 1 dirs total, 0 dirs full, 0 dirs failed I0507 06:31:41.624970 7348 tablet_bootstrap.cc:439] T 558f015eda7b4593b2bf9090cc253899 P e2ec75f7edc34d4aa7aca3a4843cf259: Bootstrap starting. I0507 06:31:41.628129 7348 tablet_bootstrap.cc:590] T 558f015eda7b4593b2bf9090cc253899 P e2ec75f7edc34d4aa7aca3a4843cf259: No blocks or log segments found. Creating new log. I0507 06:31:41.629192 7349 tablet_bootstrap.cc:439] T 558f015eda7b4593b2bf9090cc253899 P 8c31c26018bd4f6391b7f63f1af2a6a7: Bootstrap starting. I0507 06:31:41.629374 7348 log.cc:525] T 558f015eda7b4593b2bf9090cc253899 P e2ec75f7edc34d4aa7aca3a4843cf259: Log is configured to *not* fsync() on all Append() calls I0507 06:31:41.634364 7349 tablet_bootstrap.cc:590] T 558f015eda7b4593b2bf9090cc253899 P 8c31c26018bd4f6391b7f63f1af2a6a7: No blocks or log segments found. Creating new log. I0507 06:31:41.635452 7349 log.cc:525] T 558f015eda7b4593b2bf9090cc253899 P 8c31c26018bd4f6391b7f63f1af2a6a7: Log is configured to *not* fsync() on all Append() calls I0507 06:31:41.637941 7348 tablet_bootstrap.cc:439] T 558f015eda7b4593b2bf9090cc253899 P e2ec75f7edc34d4aa7aca3a4843cf259: No bootstrap required, opened a new log I0507 06:31:41.638252 7348 ts_tablet_manager.cc:1053] T 558f015eda7b4593b2bf9090cc253899 P e2ec75f7edc34d4aa7aca3a4843cf259: Time spent bootstrapping tablet: real 0.014s user 0.009s sys 0.003s I0507 06:31:41.639245 7349 tablet_bootstrap.cc:439] T 558f015eda7b4593b2bf9090cc253899 P 8c31c26018bd4f6391b7f63f1af2a6a7: No bootstrap required, opened a new log I0507 06:31:41.639551 7349 ts_tablet_manager.cc:1053] T 558f015eda7b4593b2bf9090cc253899 P 8c31c26018bd4f6391b7f63f1af2a6a7: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.000s I0507 06:31:41.641189 7352 tablet_bootstrap.cc:439] T 558f015eda7b4593b2bf9090cc253899 P 48cc028a4cfd4637a8ede1b4e421ae8c: Bootstrap starting. I0507 06:31:41.643687 7352 tablet_bootstrap.cc:590] T 558f015eda7b4593b2bf9090cc253899 P 48cc028a4cfd4637a8ede1b4e421ae8c: No blocks or log segments found. Creating new log. I0507 06:31:41.645306 7349 raft_consensus.cc:340] T 558f015eda7b4593b2bf9090cc253899 P 8c31c26018bd4f6391b7f63f1af2a6a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2ec75f7edc34d4aa7aca3a4843cf259" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 35003 } } peers { permanent_uuid: "48cc028a4cfd4637a8ede1b4e421ae8c" member_type: VOTER last_known_addr { host: "127.0.57.197" port: 40311 } } peers { permanent_uuid: "8c31c26018bd4f6391b7f63f1af2a6a7" member_type: VOTER last_known_addr { host: "127.0.57.196" port: 37823 } } I0507 06:31:41.645670 7349 raft_consensus.cc:366] T 558f015eda7b4593b2bf9090cc253899 P 8c31c26018bd4f6391b7f63f1af2a6a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0507 06:31:41.645732 7349 raft_consensus.cc:705] T 558f015eda7b4593b2bf9090cc253899 P 8c31c26018bd4f6391b7f63f1af2a6a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c31c26018bd4f6391b7f63f1af2a6a7, State: Initialized, Role: FOLLOWER I0507 06:31:41.646170 7349 consensus_queue.cc:229] T 558f015eda7b4593b2bf9090cc253899 P 8c31c26018bd4f6391b7f63f1af2a6a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2ec75f7edc34d4aa7aca3a4843cf259" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 35003 } } peers { permanent_uuid: "48cc028a4cfd4637a8ede1b4e421ae8c" member_type: VOTER last_known_addr { host: "127.0.57.197" port: 40311 } } peers { permanent_uuid: "8c31c26018bd4f6391b7f63f1af2a6a7" member_type: VOTER last_known_addr { host: "127.0.57.196" port: 37823 } } I0507 06:31:41.649061 7352 log.cc:525] T 558f015eda7b4593b2bf9090cc253899 P 48cc028a4cfd4637a8ede1b4e421ae8c: Log is configured to *not* fsync() on all Append() calls I0507 06:31:41.649130 7206 heartbeater.cc:467] Master 127.0.57.254:46549 was elected leader, sending a full tablet report... I0507 06:31:41.649471 7349 ts_tablet_manager.cc:1081] T 558f015eda7b4593b2bf9090cc253899 P 8c31c26018bd4f6391b7f63f1af2a6a7: Time spent starting tablet: real 0.010s user 0.005s sys 0.004s I0507 06:31:41.650171 7348 raft_consensus.cc:340] T 558f015eda7b4593b2bf9090cc253899 P e2ec75f7edc34d4aa7aca3a4843cf259 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2ec75f7edc34d4aa7aca3a4843cf259" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 35003 } } peers { permanent_uuid: "48cc028a4cfd4637a8ede1b4e421ae8c" member_type: VOTER last_known_addr { host: "127.0.57.197" port: 40311 } } peers { permanent_uuid: "8c31c26018bd4f6391b7f63f1af2a6a7" member_type: VOTER last_known_addr { host: "127.0.57.196" port: 37823 } } I0507 06:31:41.650545 7348 raft_consensus.cc:366] T 558f015eda7b4593b2bf9090cc253899 P e2ec75f7edc34d4aa7aca3a4843cf259 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0507 06:31:41.650626 7348 raft_consensus.cc:705] T 558f015eda7b4593b2bf9090cc253899 P e2ec75f7edc34d4aa7aca3a4843cf259 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2ec75f7edc34d4aa7aca3a4843cf259, State: Initialized, Role: FOLLOWER I0507 06:31:41.651108 7348 consensus_queue.cc:229] T 558f015eda7b4593b2bf9090cc253899 P e2ec75f7edc34d4aa7aca3a4843cf259 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2ec75f7edc34d4aa7aca3a4843cf259" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 35003 } } peers { permanent_uuid: "48cc028a4cfd4637a8ede1b4e421ae8c" member_type: VOTER last_known_addr { host: "127.0.57.197" port: 40311 } } peers { permanent_uuid: "8c31c26018bd4f6391b7f63f1af2a6a7" member_type: VOTER last_known_addr { host: "127.0.57.196" port: 37823 } } I0507 06:31:41.652019 7084 heartbeater.cc:467] Master 127.0.57.254:46549 was elected leader, sending a full tablet report... I0507 06:31:41.652742 7348 ts_tablet_manager.cc:1081] T 558f015eda7b4593b2bf9090cc253899 P e2ec75f7edc34d4aa7aca3a4843cf259: Time spent starting tablet: real 0.014s user 0.005s sys 0.004s I0507 06:31:41.652874 7352 tablet_bootstrap.cc:439] T 558f015eda7b4593b2bf9090cc253899 P 48cc028a4cfd4637a8ede1b4e421ae8c: No bootstrap required, opened a new log I0507 06:31:41.653138 7352 ts_tablet_manager.cc:1053] T 558f015eda7b4593b2bf9090cc253899 P 48cc028a4cfd4637a8ede1b4e421ae8c: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.004s I0507 06:31:41.658818 7352 raft_consensus.cc:340] T 558f015eda7b4593b2bf9090cc253899 P 48cc028a4cfd4637a8ede1b4e421ae8c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2ec75f7edc34d4aa7aca3a4843cf259" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 35003 } } peers { permanent_uuid: "48cc028a4cfd4637a8ede1b4e421ae8c" member_type: VOTER last_known_addr { host: "127.0.57.197" port: 40311 } } peers { permanent_uuid: "8c31c26018bd4f6391b7f63f1af2a6a7" member_type: VOTER last_known_addr { host: "127.0.57.196" port: 37823 } } I0507 06:31:41.659181 7352 raft_consensus.cc:366] T 558f015eda7b4593b2bf9090cc253899 P 48cc028a4cfd4637a8ede1b4e421ae8c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0507 06:31:41.659253 7352 raft_consensus.cc:705] T 558f015eda7b4593b2bf9090cc253899 P 48cc028a4cfd4637a8ede1b4e421ae8c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48cc028a4cfd4637a8ede1b4e421ae8c, State: Initialized, Role: FOLLOWER I0507 06:31:41.659729 7352 consensus_queue.cc:229] T 558f015eda7b4593b2bf9090cc253899 P 48cc028a4cfd4637a8ede1b4e421ae8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2ec75f7edc34d4aa7aca3a4843cf259" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 35003 } } peers { permanent_uuid: "48cc028a4cfd4637a8ede1b4e421ae8c" member_type: VOTER last_known_addr { host: "127.0.57.197" port: 40311 } } peers { permanent_uuid: "8c31c26018bd4f6391b7f63f1af2a6a7" member_type: VOTER last_known_addr { host: "127.0.57.196" port: 37823 } } I0507 06:31:41.660610 7328 heartbeater.cc:467] Master 127.0.57.254:46549 was elected leader, sending a full tablet report... I0507 06:31:41.660941 7352 ts_tablet_manager.cc:1081] T 558f015eda7b4593b2bf9090cc253899 P 48cc028a4cfd4637a8ede1b4e421ae8c: Time spent starting tablet: real 0.008s user 0.003s sys 0.005s W0507 06:31:41.705593 7207 tablet.cc:1786] T 558f015eda7b4593b2bf9090cc253899 P 8c31c26018bd4f6391b7f63f1af2a6a7: Can't schedule compaction. Clean time has not been advanced past its initial value. W0507 06:31:41.715174 7085 tablet.cc:1786] T 558f015eda7b4593b2bf9090cc253899 P e2ec75f7edc34d4aa7aca3a4843cf259: Can't schedule compaction. Clean time has not been advanced past its initial value. I0507 06:31:41.744634 7356 raft_consensus.cc:472] T 558f015eda7b4593b2bf9090cc253899 P 48cc028a4cfd4637a8ede1b4e421ae8c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0507 06:31:41.744856 7356 raft_consensus.cc:494] T 558f015eda7b4593b2bf9090cc253899 P 48cc028a4cfd4637a8ede1b4e421ae8c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2ec75f7edc34d4aa7aca3a4843cf259" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 35003 } } peers { permanent_uuid: "48cc028a4cfd4637a8ede1b4e421ae8c" member_type: VOTER last_known_addr { host: "127.0.57.197" port: 40311 } } peers { permanent_uuid: "8c31c26018bd4f6391b7f63f1af2a6a7" member_type: VOTER last_known_addr { host: "127.0.57.196" port: 37823 } } I0507 06:31:41.746161 7356 leader_election.cc:296] T 558f015eda7b4593b2bf9090cc253899 P 48cc028a4cfd4637a8ede1b4e421ae8c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e2ec75f7edc34d4aa7aca3a4843cf259 (127.0.57.195:35003), 8c31c26018bd4f6391b7f63f1af2a6a7 (127.0.57.196:37823) W0507 06:31:41.747373 7329 tablet.cc:1786] T 558f015eda7b4593b2bf9090cc253899 P 48cc028a4cfd4637a8ede1b4e421ae8c: Can't schedule compaction. Clean time has not been advanced past its initial value. I0507 06:31:41.753176 7041 tablet_service.cc:1286] Received RequestConsensusVote() RPC: tablet_id: "558f015eda7b4593b2bf9090cc253899" candidate_uuid: "48cc028a4cfd4637a8ede1b4e421ae8c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2ec75f7edc34d4aa7aca3a4843cf259" is_pre_election: true I0507 06:31:41.753669 7041 raft_consensus.cc:2346] T 558f015eda7b4593b2bf9090cc253899 P e2ec75f7edc34d4aa7aca3a4843cf259 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48cc028a4cfd4637a8ede1b4e421ae8c in term 0. I0507 06:31:41.754402 7218 leader_election.cc:310] T 558f015eda7b4593b2bf9090cc253899 P 48cc028a4cfd4637a8ede1b4e421ae8c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48cc028a4cfd4637a8ede1b4e421ae8c, e2ec75f7edc34d4aa7aca3a4843cf259; no voters: I0507 06:31:41.754499 7163 tablet_service.cc:1286] Received RequestConsensusVote() RPC: tablet_id: "558f015eda7b4593b2bf9090cc253899" candidate_uuid: "48cc028a4cfd4637a8ede1b4e421ae8c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c31c26018bd4f6391b7f63f1af2a6a7" is_pre_election: true I0507 06:31:41.754817 7163 raft_consensus.cc:2346] T 558f015eda7b4593b2bf9090cc253899 P 8c31c26018bd4f6391b7f63f1af2a6a7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48cc028a4cfd4637a8ede1b4e421ae8c in term 0. I0507 06:31:41.754873 7356 raft_consensus.cc:2640] T 558f015eda7b4593b2bf9090cc253899 P 48cc028a4cfd4637a8ede1b4e421ae8c [term 0 FOLLOWER]: Leader pre-election won for term 1 I0507 06:31:41.754945 7356 raft_consensus.cc:472] T 558f015eda7b4593b2bf9090cc253899 P 48cc028a4cfd4637a8ede1b4e421ae8c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0507 06:31:41.754997 7356 raft_consensus.cc:2886] T 558f015eda7b4593b2bf9090cc253899 P 48cc028a4cfd4637a8ede1b4e421ae8c [term 0 FOLLOWER]: Advancing to term 1 I0507 06:31:41.757115 7356 raft_consensus.cc:494] T 558f015eda7b4593b2bf9090cc253899 P 48cc028a4cfd4637a8ede1b4e421ae8c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2ec75f7edc34d4aa7aca3a4843cf259" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 35003 } } peers { permanent_uuid: "48cc028a4cfd4637a8ede1b4e421ae8c" member_type: VOTER last_known_addr { host: "127.0.57.197" port: 40311 } } peers { permanent_uuid: "8c31c26018bd4f6391b7f63f1af2a6a7" member_type: VOTER last_known_addr { host: "127.0.57.196" port: 37823 } } I0507 06:31:41.757719 7356 leader_election.cc:296] T 558f015eda7b4593b2bf9090cc253899 P 48cc028a4cfd4637a8ede1b4e421ae8c [CANDIDATE]: Term 1 election: Requested vote from peers e2ec75f7edc34d4aa7aca3a4843cf259 (127.0.57.195:35003), 8c31c26018bd4f6391b7f63f1af2a6a7 (127.0.57.196:37823) I0507 06:31:41.758122 7041 tablet_service.cc:1286] Received RequestConsensusVote() RPC: tablet_id: "558f015eda7b4593b2bf9090cc253899" candidate_uuid: "48cc028a4cfd4637a8ede1b4e421ae8c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2ec75f7edc34d4aa7aca3a4843cf259" I0507 06:31:41.758184 7163 tablet_service.cc:1286] Received RequestConsensusVote() RPC: tablet_id: "558f015eda7b4593b2bf9090cc253899" candidate_uuid: "48cc028a4cfd4637a8ede1b4e421ae8c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c31c26018bd4f6391b7f63f1af2a6a7" I0507 06:31:41.758266 7041 raft_consensus.cc:2886] T 558f015eda7b4593b2bf9090cc253899 P e2ec75f7edc34d4aa7aca3a4843cf259 [term 0 FOLLOWER]: Advancing to term 1 I0507 06:31:41.758358 7163 raft_consensus.cc:2886] T 558f015eda7b4593b2bf9090cc253899 P 8c31c26018bd4f6391b7f63f1af2a6a7 [term 0 FOLLOWER]: Advancing to term 1 I0507 06:31:41.760133 7041 raft_consensus.cc:2346] T 558f015eda7b4593b2bf9090cc253899 P e2ec75f7edc34d4aa7aca3a4843cf259 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48cc028a4cfd4637a8ede1b4e421ae8c in term 1. I0507 06:31:41.760210 7163 raft_consensus.cc:2346] T 558f015eda7b4593b2bf9090cc253899 P 8c31c26018bd4f6391b7f63f1af2a6a7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48cc028a4cfd4637a8ede1b4e421ae8c in term 1. I0507 06:31:41.760514 7218 leader_election.cc:310] T 558f015eda7b4593b2bf9090cc253899 P 48cc028a4cfd4637a8ede1b4e421ae8c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48cc028a4cfd4637a8ede1b4e421ae8c, e2ec75f7edc34d4aa7aca3a4843cf259; no voters: I0507 06:31:41.760891 7356 raft_consensus.cc:2640] T 558f015eda7b4593b2bf9090cc253899 P 48cc028a4cfd4637a8ede1b4e421ae8c [term 1 FOLLOWER]: Leader election won for term 1 I0507 06:31:41.761425 7356 raft_consensus.cc:667] T 558f015eda7b4593b2bf9090cc253899 P 48cc028a4cfd4637a8ede1b4e421ae8c [term 1 LEADER]: Becoming Leader. State: Replica: 48cc028a4cfd4637a8ede1b4e421ae8c, State: Running, Role: LEADER I0507 06:31:41.761833 7356 consensus_queue.cc:206] T 558f015eda7b4593b2bf9090cc253899 P 48cc028a4cfd4637a8ede1b4e421ae8c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2ec75f7edc34d4aa7aca3a4843cf259" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 35003 } } peers { permanent_uuid: "48cc028a4cfd4637a8ede1b4e421ae8c" member_type: VOTER last_known_addr { host: "127.0.57.197" port: 40311 } } peers { permanent_uuid: "8c31c26018bd4f6391b7f63f1af2a6a7" member_type: VOTER last_known_addr { host: "127.0.57.196" port: 37823 } } I0507 06:31:41.766340 6675 catalog_manager.cc:3988] T 558f015eda7b4593b2bf9090cc253899 P 48cc028a4cfd4637a8ede1b4e421ae8c reported cstate change: term changed from 0 to 1, leader changed from to 48cc028a4cfd4637a8ede1b4e421ae8c (127.0.57.197). New cstate: current_term: 1 leader_uuid: "48cc028a4cfd4637a8ede1b4e421ae8c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2ec75f7edc34d4aa7aca3a4843cf259" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 35003 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "48cc028a4cfd4637a8ede1b4e421ae8c" member_type: VOTER last_known_addr { host: "127.0.57.197" port: 40311 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8c31c26018bd4f6391b7f63f1af2a6a7" member_type: VOTER last_known_addr { host: "127.0.57.196" port: 37823 } health_report { overall_health: UNKNOWN } } } I0507 06:31:41.778036 6962 heartbeater.cc:467] Master 127.0.57.254:46549 was elected leader, sending a full tablet report... I0507 06:31:41.804445 231 external_mini_cluster.cc:623] 5 TS(s) registered with all masters I0507 06:31:41.806169 231 ts_itest-base.cc:232] Waiting for 0 tablets on tserver 18b5d1299d9146ccabb7d4819ec95db1 to finish bootstrapping I0507 06:31:41.813851 231 ts_itest-base.cc:232] Waiting for 0 tablets on tserver e84d1eaf01124e6dbc9be1f1fde546b7 to finish bootstrapping I0507 06:31:41.821233 231 ts_itest-base.cc:232] Waiting for 1 tablets on tserver e2ec75f7edc34d4aa7aca3a4843cf259 to finish bootstrapping I0507 06:31:41.827472 231 ts_itest-base.cc:232] Waiting for 1 tablets on tserver 8c31c26018bd4f6391b7f63f1af2a6a7 to finish bootstrapping I0507 06:31:41.833501 231 ts_itest-base.cc:232] Waiting for 1 tablets on tserver 48cc028a4cfd4637a8ede1b4e421ae8c to finish bootstrapping I0507 06:31:41.840386 6836 generic_service.cc:148] {username='slave'} at 127.0.0.1:58306 changed flags via RPC: tablet_copy_fault_crash_on_fetch_all from '0' to '1.0' I0507 06:31:41.841118 231 external_mini_cluster.cc:1172] Attempting to check leaks for /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver pid 6843 I0507 06:31:41.841912 6959 generic_service.cc:171] Checking for leaks (request via RPC) I0507 06:31:41.888260 231 external_mini_cluster.cc:1131] Killing /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver with pid 6843 I0507 06:31:42.153451 7041 raft_consensus.cc:1181] T 558f015eda7b4593b2bf9090cc253899 P e2ec75f7edc34d4aa7aca3a4843cf259 [term 1 FOLLOWER]: Refusing update from remote peer 48cc028a4cfd4637a8ede1b4e421ae8c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch) I0507 06:31:42.154490 7356 consensus_queue.cc:984] T 558f015eda7b4593b2bf9090cc253899 P 48cc028a4cfd4637a8ede1b4e421ae8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2ec75f7edc34d4aa7aca3a4843cf259" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 35003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0507 06:31:42.164949 7163 raft_consensus.cc:1181] T 558f015eda7b4593b2bf9090cc253899 P 8c31c26018bd4f6391b7f63f1af2a6a7 [term 1 FOLLOWER]: Refusing update from remote peer 48cc028a4cfd4637a8ede1b4e421ae8c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch) I0507 06:31:42.165815 7356 consensus_queue.cc:984] T 558f015eda7b4593b2bf9090cc253899 P 48cc028a4cfd4637a8ede1b4e421ae8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c31c26018bd4f6391b7f63f1af2a6a7" member_type: VOTER last_known_addr { host: "127.0.57.196" port: 37823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0507 06:31:46.935505 231 test_util.cc:212] Using random seed: 1594293944 I0507 06:31:46.950801 231 test_workload.cc:306] TestWorkload: Skipping table creation because table TestTable already exists I0507 06:31:46.952193 231 raft_consensus-itest-base.cc:271] Waiting until we've written at least 4MB... I0507 06:31:47.033296 7085 maintenance_manager.cc:306] P e2ec75f7edc34d4aa7aca3a4843cf259: Scheduling FlushMRSOp(558f015eda7b4593b2bf9090cc253899): 1840788 bytes log retention I0507 06:31:47.067982 7329 maintenance_manager.cc:306] P 48cc028a4cfd4637a8ede1b4e421ae8c: Scheduling FlushMRSOp(558f015eda7b4593b2bf9090cc253899): 2761146 bytes log retention I0507 06:31:47.094452 6979 maintenance_manager.cc:501] P e2ec75f7edc34d4aa7aca3a4843cf259: FlushMRSOp(558f015eda7b4593b2bf9090cc253899) complete. Timing: real 0.060s user 0.038s sys 0.004s Metrics: {"bytes_written":135803,"cfile_init":1,"compiler_manager_pool.queue_time_us":1111,"data dirs.queue_time_us":512,"data dirs.run_cpu_time_us":196,"data dirs.run_wall_time_us":1719,"drs_written":1,"lbm_read_time_us":67,"lbm_reads_lt_1ms":4,"lbm_write_time_us":557,"lbm_writes_lt_1ms":23,"rows_written":16,"thread_start_us":1183,"threads_started":2} I0507 06:31:47.095965 7085 maintenance_manager.cc:306] P e2ec75f7edc34d4aa7aca3a4843cf259: Scheduling LogGCOp(558f015eda7b4593b2bf9090cc253899): free 1840788 bytes of WAL I0507 06:31:47.096279 6979 log_reader.cc:377] T 558f015eda7b4593b2bf9090cc253899: removed 2 log segments from log reader I0507 06:31:47.096375 6979 log.cc:863] T 558f015eda7b4593b2bf9090cc253899 P e2ec75f7edc34d4aa7aca3a4843cf259: Deleting log segment in path: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.FailedTabletCopy.1557210500371863-231/raft_consensus-itest-cluster/ts-2/wal/wals/558f015eda7b4593b2bf9090cc253899/wal-000000001 (ops 1-8) I0507 06:31:47.096498 6979 log.cc:863] T 558f015eda7b4593b2bf9090cc253899 P e2ec75f7edc34d4aa7aca3a4843cf259: Deleting log segment in path: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.FailedTabletCopy.1557210500371863-231/raft_consensus-itest-cluster/ts-2/wal/wals/558f015eda7b4593b2bf9090cc253899/wal-000000002 (ops 9-15) I0507 06:31:47.097035 6979 maintenance_manager.cc:501] P e2ec75f7edc34d4aa7aca3a4843cf259: LogGCOp(558f015eda7b4593b2bf9090cc253899) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {} I0507 06:31:47.099557 7085 maintenance_manager.cc:306] P e2ec75f7edc34d4aa7aca3a4843cf259: Scheduling FlushMRSOp(558f015eda7b4593b2bf9090cc253899): 1577714 bytes log retention I0507 06:31:47.118870 231 raft_consensus-itest-base.cc:277] Waiting for log GC on 48cc028a4cfd4637a8ede1b4e421ae8c I0507 06:31:47.135406 7223 maintenance_manager.cc:501] P 48cc028a4cfd4637a8ede1b4e421ae8c: FlushMRSOp(558f015eda7b4593b2bf9090cc253899) complete. Timing: real 0.067s user 0.046s sys 0.010s Metrics: {"bytes_written":135937,"cfile_init":1,"compiler_manager_pool.queue_time_us":6915,"data dirs.queue_time_us":650,"data dirs.run_cpu_time_us":191,"data dirs.run_wall_time_us":1780,"drs_written":1,"lbm_read_time_us":64,"lbm_reads_lt_1ms":4,"lbm_write_time_us":551,"lbm_writes_lt_1ms":23,"rows_written":25,"spinlock_wait_cycles":36864,"thread_start_us":9186,"threads_started":2} I0507 06:31:47.136670 7329 maintenance_manager.cc:306] P 48cc028a4cfd4637a8ede1b4e421ae8c: Scheduling LogGCOp(558f015eda7b4593b2bf9090cc253899): free 2761146 bytes of WAL I0507 06:31:47.136950 7223 log_reader.cc:377] T 558f015eda7b4593b2bf9090cc253899: removed 3 log segments from log reader I0507 06:31:47.137032 7223 log.cc:863] T 558f015eda7b4593b2bf9090cc253899 P 48cc028a4cfd4637a8ede1b4e421ae8c: Deleting log segment in path: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.FailedTabletCopy.1557210500371863-231/raft_consensus-itest-cluster/ts-4/wal/wals/558f015eda7b4593b2bf9090cc253899/wal-000000001 (ops 1-8) I0507 06:31:47.137105 7223 log.cc:863] T 558f015eda7b4593b2bf9090cc253899 P 48cc028a4cfd4637a8ede1b4e421ae8c: Deleting log segment in path: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.FailedTabletCopy.1557210500371863-231/raft_consensus-itest-cluster/ts-4/wal/wals/558f015eda7b4593b2bf9090cc253899/wal-000000002 (ops 9-15) I0507 06:31:47.137212 7223 log.cc:863] T 558f015eda7b4593b2bf9090cc253899 P 48cc028a4cfd4637a8ede1b4e421ae8c: Deleting log segment in path: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.FailedTabletCopy.1557210500371863-231/raft_consensus-itest-cluster/ts-4/wal/wals/558f015eda7b4593b2bf9090cc253899/wal-000000003 (ops 16-22) I0507 06:31:47.137840 7223 maintenance_manager.cc:501] P 48cc028a4cfd4637a8ede1b4e421ae8c: LogGCOp(558f015eda7b4593b2bf9090cc253899) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {} I0507 06:31:47.138329 7329 maintenance_manager.cc:306] P 48cc028a4cfd4637a8ede1b4e421ae8c: Scheduling FlushMRSOp(558f015eda7b4593b2bf9090cc253899): 1840816 bytes log retention I0507 06:31:47.139771 231 raft_consensus-itest-base.cc:282] Log GC complete on 48cc028a4cfd4637a8ede1b4e421ae8c I0507 06:31:47.150789 6979 maintenance_manager.cc:501] P e2ec75f7edc34d4aa7aca3a4843cf259: FlushMRSOp(558f015eda7b4593b2bf9090cc253899) complete. Timing: real 0.050s user 0.022s sys 0.000s Metrics: {"bytes_written":135803,"cfile_init":1,"data dirs.queue_time_us":81,"data dirs.run_cpu_time_us":137,"data dirs.run_wall_time_us":498,"drs_written":1,"lbm_read_time_us":49,"lbm_reads_lt_1ms":4,"lbm_write_time_us":532,"lbm_writes_lt_1ms":23,"rows_written":15} I0507 06:31:47.157466 7085 maintenance_manager.cc:306] P e2ec75f7edc34d4aa7aca3a4843cf259: Scheduling LogGCOp(558f015eda7b4593b2bf9090cc253899): free 788819 bytes of WAL I0507 06:31:47.158062 6979 log_reader.cc:377] T 558f015eda7b4593b2bf9090cc253899: removed 1 log segments from log reader I0507 06:31:47.158159 6979 log.cc:863] T 558f015eda7b4593b2bf9090cc253899 P e2ec75f7edc34d4aa7aca3a4843cf259: Deleting log segment in path: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.FailedTabletCopy.1557210500371863-231/raft_consensus-itest-cluster/ts-2/wal/wals/558f015eda7b4593b2bf9090cc253899/wal-000000003 (ops 16-21) I0507 06:31:47.158525 6979 maintenance_manager.cc:501] P e2ec75f7edc34d4aa7aca3a4843cf259: LogGCOp(558f015eda7b4593b2bf9090cc253899) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {} I0507 06:31:47.158885 7085 maintenance_manager.cc:306] P e2ec75f7edc34d4aa7aca3a4843cf259: Scheduling CompactRowSetsOp(558f015eda7b4593b2bf9090cc253899): perf score=0.879820 I0507 06:31:47.162235 7223 maintenance_manager.cc:501] P 48cc028a4cfd4637a8ede1b4e421ae8c: FlushMRSOp(558f015eda7b4593b2bf9090cc253899) complete. Timing: real 0.024s user 0.012s sys 0.009s Metrics: {"bytes_written":135803,"cfile_init":1,"data dirs.queue_time_us":209,"data dirs.run_cpu_time_us":114,"data dirs.run_wall_time_us":480,"drs_written":1,"lbm_read_time_us":50,"lbm_reads_lt_1ms":4,"lbm_write_time_us":749,"lbm_writes_lt_1ms":23,"rows_written":11} I0507 06:31:47.163414 7329 maintenance_manager.cc:306] P 48cc028a4cfd4637a8ede1b4e421ae8c: Scheduling UndoDeltaBlockGCOp(558f015eda7b4593b2bf9090cc253899): 682 bytes on disk I0507 06:31:47.164518 7223 maintenance_manager.cc:501] P 48cc028a4cfd4637a8ede1b4e421ae8c: UndoDeltaBlockGCOp(558f015eda7b4593b2bf9090cc253899) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":2,"lbm_read_time_us":112,"lbm_reads_lt_1ms":8} I0507 06:31:47.164888 7329 maintenance_manager.cc:306] P 48cc028a4cfd4637a8ede1b4e421ae8c: Scheduling CompactRowSetsOp(558f015eda7b4593b2bf9090cc253899): perf score=0.994075 I0507 06:31:47.248194 7223 maintenance_manager.cc:501] P 48cc028a4cfd4637a8ede1b4e421ae8c: CompactRowSetsOp(558f015eda7b4593b2bf9090cc253899) complete. Timing: real 0.083s user 0.057s sys 0.024s Metrics: {"bytes_written":136001,"cfile_cache_hit":10,"cfile_cache_hit_bytes":262394,"cfile_cache_miss":26,"cfile_cache_miss_bytes":264436,"cfile_init":5,"data dirs.queue_time_us":203,"data dirs.run_cpu_time_us":322,"data dirs.run_wall_time_us":764,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":853,"lbm_reads_lt_1ms":46,"lbm_write_time_us":512,"lbm_writes_lt_1ms":23,"num_input_rowsets":2,"rows_written":36} I0507 06:31:47.249899 7329 maintenance_manager.cc:306] P 48cc028a4cfd4637a8ede1b4e421ae8c: Scheduling UndoDeltaBlockGCOp(558f015eda7b4593b2bf9090cc253899): 536 bytes on disk I0507 06:31:47.250397 7223 maintenance_manager.cc:501] P 48cc028a4cfd4637a8ede1b4e421ae8c: UndoDeltaBlockGCOp(558f015eda7b4593b2bf9090cc253899) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":54,"lbm_reads_lt_1ms":4} I0507 06:31:47.253201 6979 maintenance_manager.cc:501] P e2ec75f7edc34d4aa7aca3a4843cf259: CompactRowSetsOp(558f015eda7b4593b2bf9090cc253899) complete. Timing: real 0.094s user 0.052s sys 0.039s Metrics: {"bytes_written":135937,"cfile_cache_hit":10,"cfile_cache_hit_bytes":262392,"cfile_cache_miss":26,"cfile_cache_miss_bytes":264058,"cfile_init":7,"data dirs.queue_time_us":542,"data dirs.run_cpu_time_us":268,"data dirs.run_wall_time_us":845,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":1207,"lbm_reads_lt_1ms":54,"lbm_write_time_us":792,"lbm_writes_lt_1ms":23,"num_input_rowsets":2,"rows_written":31} I0507 06:31:47.253958 7085 maintenance_manager.cc:306] P e2ec75f7edc34d4aa7aca3a4843cf259: Scheduling UndoDeltaBlockGCOp(558f015eda7b4593b2bf9090cc253899): 481 bytes on disk I0507 06:31:47.254545 6979 maintenance_manager.cc:501] P e2ec75f7edc34d4aa7aca3a4843cf259: UndoDeltaBlockGCOp(558f015eda7b4593b2bf9090cc253899) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":54,"lbm_reads_lt_1ms":4} I0507 06:31:49.141824 231 raft_consensus-itest-base.cc:298] Servers converged with original term 1 I0507 06:31:49.141927 231 raft_consensus-itest-base.cc:303] Resuming 8c31c26018bd4f6391b7f63f1af2a6a7 I0507 06:31:49.145946 7203 generic_service.cc:148] {username='slave'} at 127.0.0.1:59364 changed flags via RPC: tablet_copy_fault_crash_on_fetch_all from '0' to '1.0' I0507 06:31:49.148481 7407 raft_consensus.cc:472] T 558f015eda7b4593b2bf9090cc253899 P 8c31c26018bd4f6391b7f63f1af2a6a7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 48cc028a4cfd4637a8ede1b4e421ae8c) I0507 06:31:49.149500 7407 raft_consensus.cc:494] T 558f015eda7b4593b2bf9090cc253899 P 8c31c26018bd4f6391b7f63f1af2a6a7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2ec75f7edc34d4aa7aca3a4843cf259" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 35003 } } peers { permanent_uuid: "48cc028a4cfd4637a8ede1b4e421ae8c" member_type: VOTER last_known_addr { host: "127.0.57.197" port: 40311 } } peers { permanent_uuid: "8c31c26018bd4f6391b7f63f1af2a6a7" member_type: VOTER last_known_addr { host: "127.0.57.196" port: 37823 } } I0507 06:31:49.150650 7407 leader_election.cc:296] T 558f015eda7b4593b2bf9090cc253899 P 8c31c26018bd4f6391b7f63f1af2a6a7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e2ec75f7edc34d4aa7aca3a4843cf259 (127.0.57.195:35003), 48cc028a4cfd4637a8ede1b4e421ae8c (127.0.57.197:40311) I0507 06:31:49.159247 7040 tablet_service.cc:1286] Received RequestConsensusVote() RPC: tablet_id: "558f015eda7b4593b2bf9090cc253899" candidate_uuid: "8c31c26018bd4f6391b7f63f1af2a6a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e2ec75f7edc34d4aa7aca3a4843cf259" is_pre_election: true I0507 06:31:49.159569 7040 raft_consensus.cc:2307] T 558f015eda7b4593b2bf9090cc253899 P e2ec75f7edc34d4aa7aca3a4843cf259 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8c31c26018bd4f6391b7f63f1af2a6a7 for term 2 because replica is either leader or believes a valid leader to be alive. I0507 06:31:49.161489 7406 consensus_queue.cc:689] T 558f015eda7b4593b2bf9090cc253899 P 48cc028a4cfd4637a8ede1b4e421ae8c [LEADER]: The logs necessary to catch up peer 8c31c26018bd4f6391b7f63f1af2a6a7 have been garbage collected. The follower will never be able to catch up (Not found: Failed to read ops 3..31: Segment 1 which contained index 3 has been GCed) I0507 06:31:49.161602 7406 consensus_queue.cc:544] T 558f015eda7b4593b2bf9090cc253899 P 48cc028a4cfd4637a8ede1b4e421ae8c [LEADER]: The logs necessary to catch up peer 8c31c26018bd4f6391b7f63f1af2a6a7 have been garbage collected. The replica will never be able to catch up I0507 06:31:49.165040 7285 tablet_service.cc:1286] Received RequestConsensusVote() RPC: tablet_id: "558f015eda7b4593b2bf9090cc253899" candidate_uuid: "8c31c26018bd4f6391b7f63f1af2a6a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "48cc028a4cfd4637a8ede1b4e421ae8c" is_pre_election: true I0507 06:31:49.165380 7285 raft_consensus.cc:2307] T 558f015eda7b4593b2bf9090cc253899 P 48cc028a4cfd4637a8ede1b4e421ae8c [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate 8c31c26018bd4f6391b7f63f1af2a6a7 for term 2 because replica is either leader or believes a valid leader to be alive. I0507 06:31:49.165922 6673 catalog_manager.cc:3739] Sending ChangeConfig:REMOVE_PEER on tablet 558f015eda7b4593b2bf9090cc253899 (attempt 1) I0507 06:31:49.165938 7096 leader_election.cc:310] T 558f015eda7b4593b2bf9090cc253899 P 8c31c26018bd4f6391b7f63f1af2a6a7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c31c26018bd4f6391b7f63f1af2a6a7; no voters: 48cc028a4cfd4637a8ede1b4e421ae8c, e2ec75f7edc34d4aa7aca3a4843cf259 I0507 06:31:49.166647 7407 raft_consensus.cc:2588] T 558f015eda7b4593b2bf9090cc253899 P 8c31c26018bd4f6391b7f63f1af2a6a7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority I0507 06:31:49.167587 7285 consensus_queue.cc:206] T 558f015eda7b4593b2bf9090cc253899 P 48cc028a4cfd4637a8ede1b4e421ae8c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 37, Committed index: 37, Last appended: 1.37, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 38 OBSOLETE_local: false peers { permanent_uuid: "e2ec75f7edc34d4aa7aca3a4843cf259" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 35003 } } peers { permanent_uuid: "48cc028a4cfd4637a8ede1b4e421ae8c" member_type: VOTER last_known_addr { host: "127.0.57.197" port: 40311 } } I0507 06:31:49.173110 7040 raft_consensus.cc:1181] T 558f015eda7b4593b2bf9090cc253899 P e2ec75f7edc34d4aa7aca3a4843cf259 [term 1 FOLLOWER]: Refusing update from remote peer 48cc028a4cfd4637a8ede1b4e421ae8c: Log matching property violated. Preceding OpId in replica: term: 1 index: 37. Preceding OpId from leader: term: 1 index: 38. (index mismatch) I0507 06:31:49.173629 7414 consensus_queue.cc:984] T 558f015eda7b4593b2bf9090cc253899 P 48cc028a4cfd4637a8ede1b4e421ae8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2ec75f7edc34d4aa7aca3a4843cf259" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 35003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 38, Last known committed idx: 37, Time since last communication: 0.000s I0507 06:31:49.181077 7406 raft_consensus.cc:2788] T 558f015eda7b4593b2bf9090cc253899 P 48cc028a4cfd4637a8ede1b4e421ae8c [term 1 LEADER]: Committing config change with OpId 1.38: config changed from index -1 to 38, VOTER 8c31c26018bd4f6391b7f63f1af2a6a7 (127.0.57.196) evicted. New config: { opid_index: 38 OBSOLETE_local: false peers { permanent_uuid: "e2ec75f7edc34d4aa7aca3a4843cf259" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 35003 } } peers { permanent_uuid: "48cc028a4cfd4637a8ede1b4e421ae8c" member_type: VOTER last_known_addr { host: "127.0.57.197" port: 40311 } } } I0507 06:31:49.181627 7040 raft_consensus.cc:2788] T 558f015eda7b4593b2bf9090cc253899 P e2ec75f7edc34d4aa7aca3a4843cf259 [term 1 FOLLOWER]: Committing config change with OpId 1.38: config changed from index -1 to 38, VOTER 8c31c26018bd4f6391b7f63f1af2a6a7 (127.0.57.196) evicted. New config: { opid_index: 38 OBSOLETE_local: false peers { permanent_uuid: "e2ec75f7edc34d4aa7aca3a4843cf259" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 35003 } } peers { permanent_uuid: "48cc028a4cfd4637a8ede1b4e421ae8c" member_type: VOTER last_known_addr { host: "127.0.57.197" port: 40311 } } } I0507 06:31:49.184820 6659 catalog_manager.cc:3531] ChangeConfig:REMOVE_PEER RPC for tablet 558f015eda7b4593b2bf9090cc253899 with cas_config_opid_index -1: ChangeConfig:REMOVE_PEER succeeded (attempt 1) I0507 06:31:49.186926 6673 catalog_manager.cc:3988] T 558f015eda7b4593b2bf9090cc253899 P e2ec75f7edc34d4aa7aca3a4843cf259 reported cstate change: config changed from index -1 to 38, VOTER 8c31c26018bd4f6391b7f63f1af2a6a7 (127.0.57.196) evicted. New cstate: current_term: 1 leader_uuid: "48cc028a4cfd4637a8ede1b4e421ae8c" committed_config { opid_index: 38 OBSOLETE_local: false peers { permanent_uuid: "e2ec75f7edc34d4aa7aca3a4843cf259" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 35003 } } peers { permanent_uuid: "48cc028a4cfd4637a8ede1b4e421ae8c" member_type: VOTER last_known_addr { host: "127.0.57.197" port: 40311 } } } I0507 06:31:49.190567 6673 catalog_manager.cc:3390] Sending DeleteTablet(TABLET_DATA_TOMBSTONED) for tablet 558f015eda7b4593b2bf9090cc253899 on 8c31c26018bd4f6391b7f63f1af2a6a7 (127.0.57.196:37823) (TS 8c31c26018bd4f6391b7f63f1af2a6a7 not found in new config with opid_index 38) I0507 06:31:49.190832 6674 catalog_manager.cc:3616] Sending ChangeConfig:ADD_PEER:NON_VOTER on tablet 558f015eda7b4593b2bf9090cc253899 (attempt 1) I0507 06:31:49.191542 7143 tablet_service.cc:1057] Processing DeleteTablet for tablet 558f015eda7b4593b2bf9090cc253899 with delete_type TABLET_DATA_TOMBSTONED (TS 8c31c26018bd4f6391b7f63f1af2a6a7 not found in new config with opid_index 38) from {username='slave'} at 127.0.0.1:59354 I0507 06:31:49.192725 7285 consensus_queue.cc:206] T 558f015eda7b4593b2bf9090cc253899 P 48cc028a4cfd4637a8ede1b4e421ae8c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 38, Committed index: 38, Last appended: 1.38, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 39 OBSOLETE_local: false peers { permanent_uuid: "e2ec75f7edc34d4aa7aca3a4843cf259" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 35003 } } peers { permanent_uuid: "48cc028a4cfd4637a8ede1b4e421ae8c" member_type: VOTER last_known_addr { host: "127.0.57.197" port: 40311 } } peers { permanent_uuid: "18b5d1299d9146ccabb7d4819ec95db1" member_type: NON_VOTER last_known_addr { host: "127.0.57.193" port: 35039 } attrs { promote: true } } I0507 06:31:49.193018 7422 tablet_replica.cc:264] T 558f015eda7b4593b2bf9090cc253899 P 8c31c26018bd4f6391b7f63f1af2a6a7: stopping tablet replica I0507 06:31:49.193383 7422 raft_consensus.cc:2143] T 558f015eda7b4593b2bf9090cc253899 P 8c31c26018bd4f6391b7f63f1af2a6a7 [term 1 FOLLOWER]: Raft consensus shutting down. I0507 06:31:49.193725 6674 catalog_manager.cc:3616] Sending ChangeConfig:ADD_PEER:NON_VOTER on tablet 558f015eda7b4593b2bf9090cc253899 (attempt 1) I0507 06:31:49.193730 7422 pending_rounds.cc:64] T 558f015eda7b4593b2bf9090cc253899 P 8c31c26018bd4f6391b7f63f1af2a6a7: Trying to abort 1 pending transactions. I0507 06:31:49.197532 7040 raft_consensus.cc:1181] T 558f015eda7b4593b2bf9090cc253899 P e2ec75f7edc34d4aa7aca3a4843cf259 [term 1 FOLLOWER]: Refusing update from remote peer 48cc028a4cfd4637a8ede1b4e421ae8c: Log matching property violated. Preceding OpId in replica: term: 1 index: 38. Preceding OpId from leader: term: 1 index: 39. (index mismatch) I0507 06:31:49.193804 7422 pending_rounds.cc:70] T 558f015eda7b4593b2bf9090cc253899 P 8c31c26018bd4f6391b7f63f1af2a6a7: Aborting transaction as it isn't in flight: id { term: 1 index: 2 } timestamp: 6378335055748268032 op_type: WRITE_OP write_request { tablet_id: "558f015eda7b4593b2bf9090cc253899" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true } } row_operations { rows: "\001\007\000\242\357\377rO\302\307?\000\000\000\000\000\000\000\000\000\000\002\000\000\000\000\000" indirect_data: "0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 I0507 06:31:49.198248 6659 catalog_manager.cc:3548] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 558f015eda7b4593b2bf9090cc253899 with cas_config_opid_index 38: ChangeConfig:ADD_PEER:NON_VOTER failed with leader 48cc028a4cfd4637a8ede1b4e421ae8c (127.0.57.197:40311) due to error UNKNOWN_ERROR; will retry: Illegal state: RaftConfig change currently pending. Only one is allowed at a time. Committed config: opid_index: 38 OBSOLETE_local: false peers { permanent_uuid: "e2ec75f7edc34d4aa7aca3a4843cf259" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 35003 } } peers { permanent_uuid: "48cc028a4cfd4637a8ede1b4e421ae8c" member_type: VOTER last_known_addr { host: "127.0.57.197" port: 40311 } }. Pending config: opid_index: 39 OBSOLETE_local: false peers { permanent_uuid: "e2ec75f7edc34d4aa7aca3a4843cf259" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 35003 } } peers { permanent_uuid: "48cc028a4cfd4637a8ede1b4e421ae8c" member_type: VOTER last_known_addr { host: "127.0.57.197" port: 40311 } } peers { permanent_uuid: "18b5d1299d9146ccabb7d4819ec95db1" member_type: NON_VOTER last_known_addr { host: "127.0.57.193" port: 35039 } attrs { promote: true } } I0507 06:31:49.198387 6659 catalog_manager.cc:3168] Scheduling retry of ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 558f015eda7b4593b2bf9090cc253899 with cas_config_opid_index 38 with a delay of 32 ms (attempt = 1) I0507 06:31:49.198817 7422 raft_consensus.cc:2170] T 558f015eda7b4593b2bf9090cc253899 P 8c31c26018bd4f6391b7f63f1af2a6a7 [term 1 FOLLOWER]: Raft consensus is shut down! I0507 06:31:49.198776 7414 consensus_queue.cc:984] T 558f015eda7b4593b2bf9090cc253899 P 48cc028a4cfd4637a8ede1b4e421ae8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2ec75f7edc34d4aa7aca3a4843cf259" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 35003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 39, Last known committed idx: 38, Time since last communication: 0.000s I0507 06:31:49.200829 7422 ts_tablet_manager.cc:1377] T 558f015eda7b4593b2bf9090cc253899 P 8c31c26018bd4f6391b7f63f1af2a6a7: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED I0507 06:31:49.201341 7414 raft_consensus.cc:2788] T 558f015eda7b4593b2bf9090cc253899 P 48cc028a4cfd4637a8ede1b4e421ae8c [term 1 LEADER]: Committing config change with OpId 1.39: config changed from index 38 to 39, NON_VOTER 18b5d1299d9146ccabb7d4819ec95db1 (127.0.57.193) added. New config: { opid_index: 39 OBSOLETE_local: false peers { permanent_uuid: "e2ec75f7edc34d4aa7aca3a4843cf259" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 35003 } } peers { permanent_uuid: "48cc028a4cfd4637a8ede1b4e421ae8c" member_type: VOTER last_known_addr { host: "127.0.57.197" port: 40311 } } peers { permanent_uuid: "18b5d1299d9146ccabb7d4819ec95db1" member_type: NON_VOTER last_known_addr { host: "127.0.57.193" port: 35039 } attrs { promote: true } } } I0507 06:31:49.202555 7040 raft_consensus.cc:2788] T 558f015eda7b4593b2bf9090cc253899 P e2ec75f7edc34d4aa7aca3a4843cf259 [term 1 FOLLOWER]: Committing config change with OpId 1.39: config changed from index 38 to 39, NON_VOTER 18b5d1299d9146ccabb7d4819ec95db1 (127.0.57.193) added. New config: { opid_index: 39 OBSOLETE_local: false peers { permanent_uuid: "e2ec75f7edc34d4aa7aca3a4843cf259" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 35003 } } peers { permanent_uuid: "48cc028a4cfd4637a8ede1b4e421ae8c" member_type: VOTER last_known_addr { host: "127.0.57.197" port: 40311 } } peers { permanent_uuid: "18b5d1299d9146ccabb7d4819ec95db1" member_type: NON_VOTER last_known_addr { host: "127.0.57.193" port: 35039 } attrs { promote: true } } } W0507 06:31:49.204552 7218 consensus_peers.cc:457] T 558f015eda7b4593b2bf9090cc253899 P 48cc028a4cfd4637a8ede1b4e421ae8c -> Peer 18b5d1299d9146ccabb7d4819ec95db1 (127.0.57.193:35039): Couldn't send request to peer 18b5d1299d9146ccabb7d4819ec95db1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 558f015eda7b4593b2bf9090cc253899. This is attempt 1: this message will repeat every 5th retry. I0507 06:31:49.205742 6659 catalog_manager.cc:3531] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 558f015eda7b4593b2bf9090cc253899 with cas_config_opid_index 38: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1) I0507 06:31:49.205971 6673 catalog_manager.cc:3988] T 558f015eda7b4593b2bf9090cc253899 P e2ec75f7edc34d4aa7aca3a4843cf259 reported cstate change: config changed from index 38 to 39, NON_VOTER 18b5d1299d9146ccabb7d4819ec95db1 (127.0.57.193) added. New cstate: current_term: 1 leader_uuid: "48cc028a4cfd4637a8ede1b4e421ae8c" committed_config { opid_index: 39 OBSOLETE_local: false peers { permanent_uuid: "e2ec75f7edc34d4aa7aca3a4843cf259" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 35003 } } peers { permanent_uuid: "48cc028a4cfd4637a8ede1b4e421ae8c" member_type: VOTER last_known_addr { host: "127.0.57.197" port: 40311 } } peers { permanent_uuid: "18b5d1299d9146ccabb7d4819ec95db1" member_type: NON_VOTER last_known_addr { host: "127.0.57.193" port: 35039 } attrs { promote: true } } } I0507 06:31:49.206455 7422 ts_tablet_manager.cc:1390] T 558f015eda7b4593b2bf9090cc253899 P 8c31c26018bd4f6391b7f63f1af2a6a7: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2 I0507 06:31:49.206554 7422 log.cc:990] T 558f015eda7b4593b2bf9090cc253899 P 8c31c26018bd4f6391b7f63f1af2a6a7: Deleting WAL directory at /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.FailedTabletCopy.1557210500371863-231/raft_consensus-itest-cluster/ts-3/wal/wals/558f015eda7b4593b2bf9090cc253899 I0507 06:31:49.208146 6657 catalog_manager.cc:3368] TS 8c31c26018bd4f6391b7f63f1af2a6a7 (127.0.57.196:37823): tablet 558f015eda7b4593b2bf9090cc253899 (table TestTable [id=d11606250418492493f264f029452e41]) successfully deleted I0507 06:31:49.220587 231 external_mini_cluster.cc:933] Running /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.FailedTabletCopy.1557210500371863-231/raft_consensus-itest-cluster/ts-1/wal --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.FailedTabletCopy.1557210500371863-231/raft_consensus-itest-cluster/ts-1/data --block_manager=log --rpc_bind_addresses=127.0.57.194:37459 --local_ip_for_outbound_sockets=127.0.57.194 --webserver_port=33367 --webserver_interface=127.0.57.194 --tserver_master_addrs=127.0.57.254:46549 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.FailedTabletCopy.1557210500371863-231/raft_consensus-itest-cluster/ts-1/logs --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.FailedTabletCopy.1557210500371863-231/raft_consensus-itest-cluster/ts-1/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=true --follower_unavailable_considered_failed_sec=10 --consensus_rpc_timeout_ms=5000 --log_compression_codec=none --log_cache_size_limit_mb=1 --log_segment_size_mb=1 --log_async_preallocate_segments=false --log_min_segments_to_retain=1 --log_max_segments_to_retain=3 --maintenance_manager_polling_interval_ms=100 --log_target_replay_size_mb=1 --max_cell_size_bytes=1000000 with env {} I0507 06:31:49.230981 6659 catalog_manager.cc:3564] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 558f015eda7b4593b2bf9090cc253899 with cas_config_opid_index 38: aborting the task: latest config opid_index 39; task opid_index 38 WARNING: Logging before InitGoogleLogging() is written to STDERR W0507 06:31:49.395215 7426 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0507 06:31:49.395439 7426 flags.cc:404] Enabled unsafe flag: --max_cell_size_bytes=1000000 W0507 06:31:49.395503 7426 flags.cc:404] Enabled unsafe flag: --never_fsync=true W0507 06:31:49.398370 7426 flags.cc:404] Enabled experimental flag: --log_compression_codec=none W0507 06:31:49.398452 7426 flags.cc:404] Enabled experimental flag: --log_max_segments_to_retain=3 W0507 06:31:49.398501 7426 flags.cc:404] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true W0507 06:31:49.398557 7426 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W0507 06:31:49.398671 7426 flags.cc:404] Enabled experimental flag: --log_target_replay_size_mb=1 W0507 06:31:49.398725 7426 flags.cc:404] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.57.194 I0507 06:31:49.402642 7426 tablet_server_main.cc:78] Tablet server non-default flags: --consensus_rpc_timeout_ms=5000 --follower_unavailable_considered_failed_sec=10 --log_compression_codec=none --log_max_segments_to_retain=3 --log_cache_size_limit_mb=1 --log_async_preallocate_segments=false --log_segment_size_mb=1 --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.FailedTabletCopy.1557210500371863-231/raft_consensus-itest-cluster/ts-1/data --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.FailedTabletCopy.1557210500371863-231/raft_consensus-itest-cluster/ts-1/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.0.57.194:37459 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.FailedTabletCopy.1557210500371863-231/raft_consensus-itest-cluster/ts-1/data/info.pb --webserver_interface=127.0.57.194 --webserver_port=33367 --max_cell_size_bytes=1000000 --tserver_master_addrs=127.0.57.254:46549 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --log_target_replay_size_mb=1 --maintenance_manager_polling_interval_ms=100 --enable_minidumps=false --local_ip_for_outbound_sockets=127.0.57.194 --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.FailedTabletCopy.1557210500371863-231/raft_consensus-itest-cluster/ts-1/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.10.0-cdh6.x-SNAPSHOT revision 7d0ee26697926e87af9ef395008799577f6ab43f build type FASTDEBUG built by jenkins at 06 May 2019 22:44:21 PST on quasar-knmpfx-1.vpc.cloudera.com build id 322017 ASAN enabled I0507 06:31:49.410179 7426 system_ntp.cc:202] NTP initialized. Skew: 500ppm Current error: 129016us I0507 06:31:49.410859 7426 fs_manager.cc:263] Metadata directory not provided I0507 06:31:49.410948 7426 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0507 06:31:49.416280 7426 fs_manager.cc:399] Time spent opening directory manager: real 0.004s user 0.005s sys 0.001s I0507 06:31:49.417603 7426 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0507 06:31:49.418201 7426 file_cache.cc:466] Constructed file cache lbm with capacity 419430 I0507 06:31:49.421691 7426 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.000s sys 0.000s I0507 06:31:49.421783 7426 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.FailedTabletCopy.1557210500371863-231/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.FailedTabletCopy.1557210500371863-231/raft_consensus-itest-cluster/ts-1/wal uuid: "e84d1eaf01124e6dbc9be1f1fde546b7" format_stamp: "Formatted at 2019-05-07 06:31:40 on dist-test-slave-dist-test-slave-rqp6" I0507 06:31:49.422777 7426 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.FailedTabletCopy.1557210500371863-231/raft_consensus-itest-cluster/ts-1/wal metadata directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.FailedTabletCopy.1557210500371863-231/raft_consensus-itest-cluster/ts-1/wal 1 data directories: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.FailedTabletCopy.1557210500371863-231/raft_consensus-itest-cluster/ts-1/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0507 06:31:49.470523 7426 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0507 06:31:49.472393 7426 ts_tablet_manager.cc:356] Loaded tablet metadata (0 total tablets, 0 live tablets) I0507 06:31:49.472482 7426 ts_tablet_manager.cc:376] Registered 0 tablets I0507 06:31:49.502631 7426 rpc_server.cc:205] RPC server started. Bound to: 127.0.57.194:37459 I0507 06:31:49.504850 7426 webserver.cc:308] Webserver started at http://127.0.57.194:33367/ using document root and password file I0507 06:31:49.505792 7426 server_base.cc:608] Dumped server information to /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.FailedTabletCopy.1557210500371863-231/raft_consensus-itest-cluster/ts-1/data/info.pb I0507 06:31:49.512876 231 external_mini_cluster.cc:995] Started /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver as pid 7426 I0507 06:31:49.522181 7545 heartbeater.cc:345] Connected to a master server at 127.0.57.254:46549 I0507 06:31:49.522418 7545 heartbeater.cc:426] Registering TS with master... I0507 06:31:49.523017 7545 heartbeater.cc:475] Master 127.0.57.254:46549 requested a full tablet report, sending... I0507 06:31:49.524242 6674 ts_manager.cc:159] Re-registered known tserver with Master: e84d1eaf01124e6dbc9be1f1fde546b7 (127.0.57.194:37459) I0507 06:31:49.525321 6674 master_service.cc:273] Signed X509 certificate for tserver {username='slave'} at 127.0.57.194:46235 I0507 06:31:49.747766 7549 ts_tablet_manager.cc:676] T 558f015eda7b4593b2bf9090cc253899 P 18b5d1299d9146ccabb7d4819ec95db1: Initiating tablet copy from peer 48cc028a4cfd4637a8ede1b4e421ae8c (127.0.57.197:40311) I0507 06:31:49.748088 7549 tablet_copy_client.cc:240] T 558f015eda7b4593b2bf9090cc253899 P 18b5d1299d9146ccabb7d4819ec95db1: tablet copy: Beginning tablet copy session from remote peer at address 127.0.57.197:40311 I0507 06:31:49.754582 7305 tablet_copy_service.cc:135] P 48cc028a4cfd4637a8ede1b4e421ae8c: Received BeginTabletCopySession request for tablet 558f015eda7b4593b2bf9090cc253899 from peer 18b5d1299d9146ccabb7d4819ec95db1 ({username='slave'} at 127.0.57.193:58931) I0507 06:31:49.754750 7305 tablet_copy_service.cc:156] P 48cc028a4cfd4637a8ede1b4e421ae8c: Beginning new tablet copy session on tablet 558f015eda7b4593b2bf9090cc253899 from peer 18b5d1299d9146ccabb7d4819ec95db1 at {username='slave'} at 127.0.57.193:58931: session id = 18b5d1299d9146ccabb7d4819ec95db1-558f015eda7b4593b2bf9090cc253899 I0507 06:31:49.757144 7305 tablet_copy_source_session.cc:202] T 558f015eda7b4593b2bf9090cc253899 P 48cc028a4cfd4637a8ede1b4e421ae8c: Tablet Copy: opened 5 blocks and 3 log segments I0507 06:31:49.760470 7549 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet 558f015eda7b4593b2bf9090cc253899. 1 dirs total, 0 dirs full, 0 dirs failed W0507 06:31:49.768779 7549 fault_injection.cc:44] FAULT INJECTION ENABLED! W0507 06:31:49.768868 7549 fault_injection.cc:45] THIS SERVER MAY CRASH! E0507 06:31:49.769004 7549 fault_injection.cc:60] injecting fault for kudu-tserver (pid 6710): FLAGS_tablet_copy_fault_crash_on_fetch_all (process will exit) I0507 06:31:50.527559 7545 heartbeater.cc:467] Master 127.0.57.254:46549 was elected leader, sending a full tablet report... W0507 06:31:52.182607 7218 consensus_peers.cc:457] T 558f015eda7b4593b2bf9090cc253899 P 48cc028a4cfd4637a8ede1b4e421ae8c -> Peer 18b5d1299d9146ccabb7d4819ec95db1 (127.0.57.193:35039): Couldn't send request to peer 18b5d1299d9146ccabb7d4819ec95db1. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.57.193:35039: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry. W0507 06:31:54.816354 7218 consensus_peers.cc:457] T 558f015eda7b4593b2bf9090cc253899 P 48cc028a4cfd4637a8ede1b4e421ae8c -> Peer 18b5d1299d9146ccabb7d4819ec95db1 (127.0.57.193:35039): Couldn't send request to peer 18b5d1299d9146ccabb7d4819ec95db1. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.57.193:35039: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry. W0507 06:31:57.138415 7218 consensus_peers.cc:457] T 558f015eda7b4593b2bf9090cc253899 P 48cc028a4cfd4637a8ede1b4e421ae8c -> Peer 18b5d1299d9146ccabb7d4819ec95db1 (127.0.57.193:35039): Couldn't send request to peer 18b5d1299d9146ccabb7d4819ec95db1. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.57.193:35039: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry. W0507 06:31:59.670851 7218 consensus_peers.cc:457] T 558f015eda7b4593b2bf9090cc253899 P 48cc028a4cfd4637a8ede1b4e421ae8c -> Peer 18b5d1299d9146ccabb7d4819ec95db1 (127.0.57.193:35039): Couldn't send request to peer 18b5d1299d9146ccabb7d4819ec95db1. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.57.193:35039: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry. I0507 06:32:00.272064 7562 consensus_queue.cc:544] T 558f015eda7b4593b2bf9090cc253899 P 48cc028a4cfd4637a8ede1b4e421ae8c [LEADER]: Leader has been unable to successfully communicate with peer 18b5d1299d9146ccabb7d4819ec95db1 for more than 10 seconds (10.502s) I0507 06:32:00.274565 6674 catalog_manager.cc:3739] Sending ChangeConfig:REMOVE_PEER on tablet 558f015eda7b4593b2bf9090cc253899 (attempt 1) I0507 06:32:00.275753 7285 consensus_queue.cc:206] T 558f015eda7b4593b2bf9090cc253899 P 48cc028a4cfd4637a8ede1b4e421ae8c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 39, Committed index: 39, Last appended: 1.39, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 40 OBSOLETE_local: false peers { permanent_uuid: "e2ec75f7edc34d4aa7aca3a4843cf259" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 35003 } } peers { permanent_uuid: "48cc028a4cfd4637a8ede1b4e421ae8c" member_type: VOTER last_known_addr { host: "127.0.57.197" port: 40311 } } I0507 06:32:00.277853 7040 raft_consensus.cc:1181] T 558f015eda7b4593b2bf9090cc253899 P e2ec75f7edc34d4aa7aca3a4843cf259 [term 1 FOLLOWER]: Refusing update from remote peer 48cc028a4cfd4637a8ede1b4e421ae8c: Log matching property violated. Preceding OpId in replica: term: 1 index: 39. Preceding OpId from leader: term: 1 index: 40. (index mismatch) I0507 06:32:00.278558 7562 consensus_queue.cc:984] T 558f015eda7b4593b2bf9090cc253899 P 48cc028a4cfd4637a8ede1b4e421ae8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2ec75f7edc34d4aa7aca3a4843cf259" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 35003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 40, Last known committed idx: 39, Time since last communication: 0.000s I0507 06:32:00.280849 7566 raft_consensus.cc:2788] T 558f015eda7b4593b2bf9090cc253899 P 48cc028a4cfd4637a8ede1b4e421ae8c [term 1 LEADER]: Committing config change with OpId 1.40: config changed from index 39 to 40, NON_VOTER 18b5d1299d9146ccabb7d4819ec95db1 (127.0.57.193) evicted. New config: { opid_index: 40 OBSOLETE_local: false peers { permanent_uuid: "e2ec75f7edc34d4aa7aca3a4843cf259" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 35003 } } peers { permanent_uuid: "48cc028a4cfd4637a8ede1b4e421ae8c" member_type: VOTER last_known_addr { host: "127.0.57.197" port: 40311 } } } I0507 06:32:00.281540 7040 raft_consensus.cc:2788] T 558f015eda7b4593b2bf9090cc253899 P e2ec75f7edc34d4aa7aca3a4843cf259 [term 1 FOLLOWER]: Committing config change with OpId 1.40: config changed from index 39 to 40, NON_VOTER 18b5d1299d9146ccabb7d4819ec95db1 (127.0.57.193) evicted. New config: { opid_index: 40 OBSOLETE_local: false peers { permanent_uuid: "e2ec75f7edc34d4aa7aca3a4843cf259" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 35003 } } peers { permanent_uuid: "48cc028a4cfd4637a8ede1b4e421ae8c" member_type: VOTER last_known_addr { host: "127.0.57.197" port: 40311 } } } I0507 06:32:00.283463 6659 catalog_manager.cc:3531] ChangeConfig:REMOVE_PEER RPC for tablet 558f015eda7b4593b2bf9090cc253899 with cas_config_opid_index 39: ChangeConfig:REMOVE_PEER succeeded (attempt 1) I0507 06:32:00.284230 6674 catalog_manager.cc:3988] T 558f015eda7b4593b2bf9090cc253899 P 48cc028a4cfd4637a8ede1b4e421ae8c reported cstate change: config changed from index 39 to 40, NON_VOTER 18b5d1299d9146ccabb7d4819ec95db1 (127.0.57.193) evicted. New cstate: current_term: 1 leader_uuid: "48cc028a4cfd4637a8ede1b4e421ae8c" committed_config { opid_index: 40 OBSOLETE_local: false peers { permanent_uuid: "e2ec75f7edc34d4aa7aca3a4843cf259" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 35003 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "48cc028a4cfd4637a8ede1b4e421ae8c" member_type: VOTER last_known_addr { host: "127.0.57.197" port: 40311 } health_report { overall_health: HEALTHY } } } I0507 06:32:00.291622 6674 catalog_manager.cc:3390] Sending DeleteTablet(TABLET_DATA_TOMBSTONED) for tablet 558f015eda7b4593b2bf9090cc253899 on 18b5d1299d9146ccabb7d4819ec95db1 (127.0.57.193:35039) (TS 18b5d1299d9146ccabb7d4819ec95db1 not found in new config with opid_index 40) I0507 06:32:00.293802 6674 catalog_manager.cc:3616] Sending ChangeConfig:ADD_PEER:NON_VOTER on tablet 558f015eda7b4593b2bf9090cc253899 (attempt 1) I0507 06:32:00.296092 6674 catalog_manager.cc:3616] Sending ChangeConfig:ADD_PEER:NON_VOTER on tablet 558f015eda7b4593b2bf9090cc253899 (attempt 1) W0507 06:32:00.298985 6659 catalog_manager.cc:3132] TS 18b5d1299d9146ccabb7d4819ec95db1 (127.0.57.193:35039): Delete Tablet RPC failed for tablet 558f015eda7b4593b2bf9090cc253899: Network error: Client connection negotiation failed: client connection to 127.0.57.193:35039: connect: Connection refused (error 111) I0507 06:32:00.299089 6659 catalog_manager.cc:3168] Scheduling retry of 558f015eda7b4593b2bf9090cc253899 Delete Tablet RPC for TS=18b5d1299d9146ccabb7d4819ec95db1 with a delay of 63 ms (attempt = 1) I0507 06:32:00.300176 7285 consensus_queue.cc:206] T 558f015eda7b4593b2bf9090cc253899 P 48cc028a4cfd4637a8ede1b4e421ae8c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 40, Committed index: 40, Last appended: 1.40, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 41 OBSOLETE_local: false peers { permanent_uuid: "e2ec75f7edc34d4aa7aca3a4843cf259" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 35003 } } peers { permanent_uuid: "48cc028a4cfd4637a8ede1b4e421ae8c" member_type: VOTER last_known_addr { host: "127.0.57.197" port: 40311 } } peers { permanent_uuid: "e84d1eaf01124e6dbc9be1f1fde546b7" member_type: NON_VOTER last_known_addr { host: "127.0.57.194" port: 37459 } attrs { promote: true } } I0507 06:32:00.302131 6659 catalog_manager.cc:3548] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 558f015eda7b4593b2bf9090cc253899 with cas_config_opid_index 40: ChangeConfig:ADD_PEER:NON_VOTER failed with leader 48cc028a4cfd4637a8ede1b4e421ae8c (127.0.57.197:40311) due to error UNKNOWN_ERROR; will retry: Illegal state: RaftConfig change currently pending. Only one is allowed at a time. Committed config: opid_index: 40 OBSOLETE_local: false peers { permanent_uuid: "e2ec75f7edc34d4aa7aca3a4843cf259" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 35003 } } peers { permanent_uuid: "48cc028a4cfd4637a8ede1b4e421ae8c" member_type: VOTER last_known_addr { host: "127.0.57.197" port: 40311 } }. Pending config: opid_index: 41 OBSOLETE_local: false peers { permanent_uuid: "e2ec75f7edc34d4aa7aca3a4843cf259" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 35003 } } peers { permanent_uuid: "48cc028a4cfd4637a8ede1b4e421ae8c" member_type: VOTER last_known_addr { host: "127.0.57.197" port: 40311 } } peers { permanent_uuid: "e84d1eaf01124e6dbc9be1f1fde546b7" member_type: NON_VOTER last_known_addr { host: "127.0.57.194" port: 37459 } attrs { promote: true } } I0507 06:32:00.302211 6659 catalog_manager.cc:3168] Scheduling retry of ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 558f015eda7b4593b2bf9090cc253899 with cas_config_opid_index 40 with a delay of 64 ms (attempt = 1) I0507 06:32:00.302487 7040 raft_consensus.cc:1181] T 558f015eda7b4593b2bf9090cc253899 P e2ec75f7edc34d4aa7aca3a4843cf259 [term 1 FOLLOWER]: Refusing update from remote peer 48cc028a4cfd4637a8ede1b4e421ae8c: Log matching property violated. Preceding OpId in replica: term: 1 index: 40. Preceding OpId from leader: term: 1 index: 41. (index mismatch) I0507 06:32:00.303035 7565 consensus_queue.cc:984] T 558f015eda7b4593b2bf9090cc253899 P 48cc028a4cfd4637a8ede1b4e421ae8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2ec75f7edc34d4aa7aca3a4843cf259" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 35003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 41, Last known committed idx: 40, Time since last communication: 0.000s I0507 06:32:00.303872 7085 maintenance_manager.cc:306] P e2ec75f7edc34d4aa7aca3a4843cf259: Scheduling LogGCOp(558f015eda7b4593b2bf9090cc253899): free 788895 bytes of WAL I0507 06:32:00.304200 6979 log_reader.cc:377] T 558f015eda7b4593b2bf9090cc253899: removed 1 log segments from log reader I0507 06:32:00.304281 6979 log.cc:863] T 558f015eda7b4593b2bf9090cc253899 P e2ec75f7edc34d4aa7aca3a4843cf259: Deleting log segment in path: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.FailedTabletCopy.1557210500371863-231/raft_consensus-itest-cluster/ts-2/wal/wals/558f015eda7b4593b2bf9090cc253899/wal-000000004 (ops 22-27) I0507 06:32:00.304618 6979 maintenance_manager.cc:501] P e2ec75f7edc34d4aa7aca3a4843cf259: LogGCOp(558f015eda7b4593b2bf9090cc253899) complete. Timing: real 0.000s user 0.000s sys 0.001s Metrics: {} I0507 06:32:00.305503 7562 raft_consensus.cc:2788] T 558f015eda7b4593b2bf9090cc253899 P 48cc028a4cfd4637a8ede1b4e421ae8c [term 1 LEADER]: Committing config change with OpId 1.41: config changed from index 40 to 41, NON_VOTER e84d1eaf01124e6dbc9be1f1fde546b7 (127.0.57.194) added. New config: { opid_index: 41 OBSOLETE_local: false peers { permanent_uuid: "e2ec75f7edc34d4aa7aca3a4843cf259" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 35003 } } peers { permanent_uuid: "48cc028a4cfd4637a8ede1b4e421ae8c" member_type: VOTER last_known_addr { host: "127.0.57.197" port: 40311 } } peers { permanent_uuid: "e84d1eaf01124e6dbc9be1f1fde546b7" member_type: NON_VOTER last_known_addr { host: "127.0.57.194" port: 37459 } attrs { promote: true } } } I0507 06:32:00.306054 7040 raft_consensus.cc:2788] T 558f015eda7b4593b2bf9090cc253899 P e2ec75f7edc34d4aa7aca3a4843cf259 [term 1 FOLLOWER]: Committing config change with OpId 1.41: config changed from index 40 to 41, NON_VOTER e84d1eaf01124e6dbc9be1f1fde546b7 (127.0.57.194) added. New config: { opid_index: 41 OBSOLETE_local: false peers { permanent_uuid: "e2ec75f7edc34d4aa7aca3a4843cf259" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 35003 } } peers { permanent_uuid: "48cc028a4cfd4637a8ede1b4e421ae8c" member_type: VOTER last_known_addr { host: "127.0.57.197" port: 40311 } } peers { permanent_uuid: "e84d1eaf01124e6dbc9be1f1fde546b7" member_type: NON_VOTER last_known_addr { host: "127.0.57.194" port: 37459 } attrs { promote: true } } } I0507 06:32:00.307848 6659 catalog_manager.cc:3531] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 558f015eda7b4593b2bf9090cc253899 with cas_config_opid_index 40: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1) I0507 06:32:00.309176 6674 catalog_manager.cc:3988] T 558f015eda7b4593b2bf9090cc253899 P 48cc028a4cfd4637a8ede1b4e421ae8c reported cstate change: config changed from index 40 to 41, NON_VOTER e84d1eaf01124e6dbc9be1f1fde546b7 (127.0.57.194) added. New cstate: current_term: 1 leader_uuid: "48cc028a4cfd4637a8ede1b4e421ae8c" committed_config { opid_index: 41 OBSOLETE_local: false peers { permanent_uuid: "e2ec75f7edc34d4aa7aca3a4843cf259" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 35003 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "48cc028a4cfd4637a8ede1b4e421ae8c" member_type: VOTER last_known_addr { host: "127.0.57.197" port: 40311 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e84d1eaf01124e6dbc9be1f1fde546b7" member_type: NON_VOTER last_known_addr { host: "127.0.57.194" port: 37459 } attrs { promote: true } health_report { overall_health: UNKNOWN } } } W0507 06:32:00.311627 7218 consensus_peers.cc:457] T 558f015eda7b4593b2bf9090cc253899 P 48cc028a4cfd4637a8ede1b4e421ae8c -> Peer e84d1eaf01124e6dbc9be1f1fde546b7 (127.0.57.194:37459): Couldn't send request to peer e84d1eaf01124e6dbc9be1f1fde546b7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 558f015eda7b4593b2bf9090cc253899. This is attempt 1: this message will repeat every 5th retry. I0507 06:32:00.362455 6659 catalog_manager.cc:3390] Sending DeleteTablet(TABLET_DATA_TOMBSTONED) for tablet 558f015eda7b4593b2bf9090cc253899 on 18b5d1299d9146ccabb7d4819ec95db1 (127.0.57.193:35039) (TS 18b5d1299d9146ccabb7d4819ec95db1 not found in new config with opid_index 40) W0507 06:32:00.363533 6659 catalog_manager.cc:3132] TS 18b5d1299d9146ccabb7d4819ec95db1 (127.0.57.193:35039): Delete Tablet RPC failed for tablet 558f015eda7b4593b2bf9090cc253899: Network error: Client connection negotiation failed: client connection to 127.0.57.193:35039: connect: Connection refused (error 111) I0507 06:32:00.363610 6659 catalog_manager.cc:3168] Scheduling retry of 558f015eda7b4593b2bf9090cc253899 Delete Tablet RPC for TS=18b5d1299d9146ccabb7d4819ec95db1 with a delay of 79 ms (attempt = 2) I0507 06:32:00.367025 6659 catalog_manager.cc:3564] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 558f015eda7b4593b2bf9090cc253899 with cas_config_opid_index 40: aborting the task: latest config opid_index 41; task opid_index 40 I0507 06:32:00.443760 6659 catalog_manager.cc:3390] Sending DeleteTablet(TABLET_DATA_TOMBSTONED) for tablet 558f015eda7b4593b2bf9090cc253899 on 18b5d1299d9146ccabb7d4819ec95db1 (127.0.57.193:35039) (TS 18b5d1299d9146ccabb7d4819ec95db1 not found in new config with opid_index 40) W0507 06:32:00.445026 6659 catalog_manager.cc:3132] TS 18b5d1299d9146ccabb7d4819ec95db1 (127.0.57.193:35039): Delete Tablet RPC failed for tablet 558f015eda7b4593b2bf9090cc253899: Network error: Client connection negotiation failed: client connection to 127.0.57.193:35039: connect: Connection refused (error 111) I0507 06:32:00.445106 6659 catalog_manager.cc:3168] Scheduling retry of 558f015eda7b4593b2bf9090cc253899 Delete Tablet RPC for TS=18b5d1299d9146ccabb7d4819ec95db1 with a delay of 86 ms (attempt = 3) I0507 06:32:00.531883 6659 catalog_manager.cc:3390] Sending DeleteTablet(TABLET_DATA_TOMBSTONED) for tablet 558f015eda7b4593b2bf9090cc253899 on 18b5d1299d9146ccabb7d4819ec95db1 (127.0.57.193:35039) (TS 18b5d1299d9146ccabb7d4819ec95db1 not found in new config with opid_index 40) W0507 06:32:00.533156 6659 catalog_manager.cc:3132] TS 18b5d1299d9146ccabb7d4819ec95db1 (127.0.57.193:35039): Delete Tablet RPC failed for tablet 558f015eda7b4593b2bf9090cc253899: Network error: Client connection negotiation failed: client connection to 127.0.57.193:35039: connect: Connection refused (error 111) I0507 06:32:00.533272 6659 catalog_manager.cc:3168] Scheduling retry of 558f015eda7b4593b2bf9090cc253899 Delete Tablet RPC for TS=18b5d1299d9146ccabb7d4819ec95db1 with a delay of 144 ms (attempt = 4) I0507 06:32:00.660985 231 external_mini_cluster.cc:1172] Attempting to check leaks for /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver pid 7426 I0507 06:32:00.668575 7542 generic_service.cc:171] Checking for leaks (request via RPC) I0507 06:32:00.678022 6659 catalog_manager.cc:3390] Sending DeleteTablet(TABLET_DATA_TOMBSTONED) for tablet 558f015eda7b4593b2bf9090cc253899 on 18b5d1299d9146ccabb7d4819ec95db1 (127.0.57.193:35039) (TS 18b5d1299d9146ccabb7d4819ec95db1 not found in new config with opid_index 40) W0507 06:32:00.679121 6659 catalog_manager.cc:3132] TS 18b5d1299d9146ccabb7d4819ec95db1 (127.0.57.193:35039): Delete Tablet RPC failed for tablet 558f015eda7b4593b2bf9090cc253899: Network error: Client connection negotiation failed: client connection to 127.0.57.193:35039: connect: Connection refused (error 111) I0507 06:32:00.679219 6659 catalog_manager.cc:3168] Scheduling retry of 558f015eda7b4593b2bf9090cc253899 Delete Tablet RPC for TS=18b5d1299d9146ccabb7d4819ec95db1 with a delay of 292 ms (attempt = 5) I0507 06:32:00.710582 231 external_mini_cluster.cc:1131] Killing /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver with pid 7426 I0507 06:32:00.720906 231 external_mini_cluster.cc:1172] Attempting to check leaks for /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver pid 6965 I0507 06:32:00.721997 7081 generic_service.cc:171] Checking for leaks (request via RPC) I0507 06:32:00.805563 231 external_mini_cluster.cc:1131] Killing /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver with pid 6965 I0507 06:32:00.822289 231 external_mini_cluster.cc:1172] Attempting to check leaks for /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver pid 7087 I0507 06:32:00.823339 7203 generic_service.cc:171] Checking for leaks (request via RPC) W0507 06:32:00.865106 7218 consensus_peers.cc:457] T 558f015eda7b4593b2bf9090cc253899 P 48cc028a4cfd4637a8ede1b4e421ae8c -> Peer e2ec75f7edc34d4aa7aca3a4843cf259 (127.0.57.195:35003): Couldn't send request to peer e2ec75f7edc34d4aa7aca3a4843cf259. Status: Network error: Client connection negotiation failed: client connection to 127.0.57.195:35003: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry. I0507 06:32:00.878098 231 external_mini_cluster.cc:1131] Killing /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver with pid 7087 I0507 06:32:00.889750 231 external_mini_cluster.cc:1172] Attempting to check leaks for /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver pid 7209 I0507 06:32:00.890822 7325 generic_service.cc:171] Checking for leaks (request via RPC) W0507 06:32:00.891496 7218 consensus_peers.cc:435] T 558f015eda7b4593b2bf9090cc253899 P 48cc028a4cfd4637a8ede1b4e421ae8c -> Peer e84d1eaf01124e6dbc9be1f1fde546b7 (127.0.57.194:37459): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.57.194:37459: connect: Connection refused (error 111) I0507 06:32:00.972198 6659 catalog_manager.cc:3390] Sending DeleteTablet(TABLET_DATA_TOMBSTONED) for tablet 558f015eda7b4593b2bf9090cc253899 on 18b5d1299d9146ccabb7d4819ec95db1 (127.0.57.193:35039) (TS 18b5d1299d9146ccabb7d4819ec95db1 not found in new config with opid_index 40) W0507 06:32:00.973332 6659 catalog_manager.cc:3132] TS 18b5d1299d9146ccabb7d4819ec95db1 (127.0.57.193:35039): Delete Tablet RPC failed for tablet 558f015eda7b4593b2bf9090cc253899: Network error: Client connection negotiation failed: client connection to 127.0.57.193:35039: connect: Connection refused (error 111) I0507 06:32:00.973421 6659 catalog_manager.cc:3168] Scheduling retry of 558f015eda7b4593b2bf9090cc253899 Delete Tablet RPC for TS=18b5d1299d9146ccabb7d4819ec95db1 with a delay of 524 ms (attempt = 6) I0507 06:32:00.976469 231 external_mini_cluster.cc:1131] Killing /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver with pid 7209 I0507 06:32:00.994619 231 external_mini_cluster.cc:1172] Attempting to check leaks for /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-master pid 6650 I0507 06:32:00.995715 6706 generic_service.cc:171] Checking for leaks (request via RPC) I0507 06:32:01.085157 231 external_mini_cluster.cc:1131] Killing /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-master with pid 6650 [ OK ] RaftConsensusNonVoterITest.FailedTabletCopy (21238 ms) [ RUN ] RaftConsensusNonVoterITest.RestartClusterWithNonVoter I0507 06:32:01.114851 231 test_util.cc:212] Using random seed: 1608473265 I0507 06:32:01.116369 231 ts_itest-base.cc:112] Starting cluster with: I0507 06:32:01.116421 231 ts_itest-base.cc:113] -------------- I0507 06:32:01.116438 231 ts_itest-base.cc:114] 7 tablet servers I0507 06:32:01.116463 231 ts_itest-base.cc:115] 3 replicas per TS I0507 06:32:01.116487 231 ts_itest-base.cc:116] -------------- I0507 06:32:01.117771 231 external_mini_cluster.cc:933] Running /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-master /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-master --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.RestartClusterWithNonVoter.1557210500371863-231/raft_consensus-itest-cluster/master-0/wal --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.RestartClusterWithNonVoter.1557210500371863-231/raft_consensus-itest-cluster/master-0/data --block_manager=log --webserver_interface=localhost --ipki_ca_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.0.57.254:34231 --webserver_interface=127.0.57.254 --webserver_port=0 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.RestartClusterWithNonVoter.1557210500371863-231/raft_consensus-itest-cluster/master-0/logs --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.RestartClusterWithNonVoter.1557210500371863-231/raft_consensus-itest-cluster/master-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=true --rpc_reuseport=true with env {} WARNING: Logging before InitGoogleLogging() is written to STDERR W0507 06:32:01.310923 7591 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0507 06:32:01.311149 7591 flags.cc:404] Enabled unsafe flag: --never_fsync=true W0507 06:32:01.313650 7591 flags.cc:404] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true W0507 06:32:01.313712 7591 flags.cc:404] Enabled experimental flag: --ipki_ca_key_size=1024 W0507 06:32:01.313741 7591 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W0507 06:32:01.313791 7591 flags.cc:404] Enabled experimental flag: --tsk_num_rsa_bits=512 W0507 06:32:01.313836 7591 flags.cc:404] Enabled experimental flag: --rpc_reuseport=true I0507 06:32:01.318348 7591 master_main.cc:73] Master server non-default flags: --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.RestartClusterWithNonVoter.1557210500371863-231/raft_consensus-itest-cluster/master-0/data --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.RestartClusterWithNonVoter.1557210500371863-231/raft_consensus-itest-cluster/master-0/wal --ipki_ca_key_size=1024 --ipki_server_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.0.57.254:34231 --rpc_reuseport=true --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.RestartClusterWithNonVoter.1557210500371863-231/raft_consensus-itest-cluster/master-0/data/info.pb --webserver_interface=127.0.57.254 --webserver_port=0 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.RestartClusterWithNonVoter.1557210500371863-231/raft_consensus-itest-cluster/master-0/logs --logbuflevel=-1 --logtostderr=true Master server version: kudu 1.10.0-cdh6.x-SNAPSHOT revision 7d0ee26697926e87af9ef395008799577f6ab43f build type FASTDEBUG built by jenkins at 06 May 2019 22:44:21 PST on quasar-knmpfx-1.vpc.cloudera.com build id 322017 ASAN enabled I0507 06:32:01.325011 7591 system_ntp.cc:202] NTP initialized. Skew: 500ppm Current error: 135016us I0507 06:32:01.326201 7591 fs_manager.cc:263] Metadata directory not provided I0507 06:32:01.326270 7591 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0507 06:32:01.326514 7591 server_base.cc:433] Could not load existing FS layout: Not found: could not find a healthy instance file I0507 06:32:01.326583 7591 server_base.cc:434] Attempting to create new FS layout instead I0507 06:32:01.329025 7591 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.RestartClusterWithNonVoter.1557210500371863-231/raft_consensus-itest-cluster/master-0/data/instance: uuid: "5fbef3ba10d6481bac975db052d4d399" format_stamp: "Formatted at 2019-05-07 06:32:01 on dist-test-slave-dist-test-slave-rqp6" I0507 06:32:01.329529 7591 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.RestartClusterWithNonVoter.1557210500371863-231/raft_consensus-itest-cluster/master-0/wal/instance: uuid: "5fbef3ba10d6481bac975db052d4d399" format_stamp: "Formatted at 2019-05-07 06:32:01 on dist-test-slave-dist-test-slave-rqp6" I0507 06:32:01.335877 7591 fs_manager.cc:503] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s I0507 06:32:01.336917 7591 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0507 06:32:01.337430 7591 file_cache.cc:466] Constructed file cache lbm with capacity 419430 I0507 06:32:01.341104 7591 fs_manager.cc:419] Time spent opening block manager: real 0.003s user 0.000s sys 0.000s I0507 06:32:01.341192 7591 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.RestartClusterWithNonVoter.1557210500371863-231/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.RestartClusterWithNonVoter.1557210500371863-231/raft_consensus-itest-cluster/master-0/wal uuid: "5fbef3ba10d6481bac975db052d4d399" format_stamp: "Formatted at 2019-05-07 06:32:01 on dist-test-slave-dist-test-slave-rqp6" I0507 06:32:01.341328 7591 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.RestartClusterWithNonVoter.1557210500371863-231/raft_consensus-itest-cluster/master-0/wal metadata directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.RestartClusterWithNonVoter.1557210500371863-231/raft_consensus-itest-cluster/master-0/wal 1 data directories: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.RestartClusterWithNonVoter.1557210500371863-231/raft_consensus-itest-cluster/master-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0507 06:32:01.403771 7591 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0507 06:32:01.417119 7591 rpc_server.cc:205] RPC server started. Bound to: 127.0.57.254:34231 I0507 06:32:01.418270 7591 webserver.cc:308] Webserver started at http://127.0.57.254:33459/ using document root and password file I0507 06:32:01.419008 7591 server_base.cc:608] Dumped server information to /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.RestartClusterWithNonVoter.1557210500371863-231/raft_consensus-itest-cluster/master-0/data/info.pb I0507 06:32:01.419391 231 external_mini_cluster.cc:995] Started /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-master as pid 7591 I0507 06:32:01.420204 231 external_mini_cluster.cc:933] Running /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.RestartClusterWithNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-0/wal --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.RestartClusterWithNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data --block_manager=log --rpc_bind_addresses=127.0.57.193:0 --local_ip_for_outbound_sockets=127.0.57.193 --webserver_interface=127.0.57.193 --webserver_port=0 --tserver_master_addrs=127.0.57.254:34231 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.RestartClusterWithNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-0/logs --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.RestartClusterWithNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=true --tablet_copy_download_file_inject_latency_ms=10000 --follower_unavailable_considered_failed_sec=2 --consensus_rpc_timeout_ms=5000 with env {} I0507 06:32:01.422411 7650 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0507 06:32:01.433351 7650 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 5fbef3ba10d6481bac975db052d4d399: Bootstrap starting. I0507 06:32:01.436043 7650 tablet_bootstrap.cc:590] T 00000000000000000000000000000000 P 5fbef3ba10d6481bac975db052d4d399: No blocks or log segments found. Creating new log. I0507 06:32:01.436982 7650 log.cc:525] T 00000000000000000000000000000000 P 5fbef3ba10d6481bac975db052d4d399: Log is configured to *not* fsync() on all Append() calls I0507 06:32:01.438633 7650 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 5fbef3ba10d6481bac975db052d4d399: No bootstrap required, opened a new log I0507 06:32:01.445016 7650 raft_consensus.cc:340] T 00000000000000000000000000000000 P 5fbef3ba10d6481bac975db052d4d399 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fbef3ba10d6481bac975db052d4d399" member_type: VOTER } I0507 06:32:01.445250 7650 raft_consensus.cc:366] T 00000000000000000000000000000000 P 5fbef3ba10d6481bac975db052d4d399 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0507 06:32:01.445307 7650 raft_consensus.cc:705] T 00000000000000000000000000000000 P 5fbef3ba10d6481bac975db052d4d399 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fbef3ba10d6481bac975db052d4d399, State: Initialized, Role: FOLLOWER I0507 06:32:01.445907 7650 consensus_queue.cc:229] T 00000000000000000000000000000000 P 5fbef3ba10d6481bac975db052d4d399 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fbef3ba10d6481bac975db052d4d399" member_type: VOTER } I0507 06:32:01.446117 7650 raft_consensus.cc:380] T 00000000000000000000000000000000 P 5fbef3ba10d6481bac975db052d4d399 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0507 06:32:01.446190 7650 raft_consensus.cc:472] T 00000000000000000000000000000000 P 5fbef3ba10d6481bac975db052d4d399 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0507 06:32:01.446278 7650 raft_consensus.cc:2886] T 00000000000000000000000000000000 P 5fbef3ba10d6481bac975db052d4d399 [term 0 FOLLOWER]: Advancing to term 1 I0507 06:32:01.447888 7650 raft_consensus.cc:494] T 00000000000000000000000000000000 P 5fbef3ba10d6481bac975db052d4d399 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fbef3ba10d6481bac975db052d4d399" member_type: VOTER } I0507 06:32:01.448305 7650 leader_election.cc:310] T 00000000000000000000000000000000 P 5fbef3ba10d6481bac975db052d4d399 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5fbef3ba10d6481bac975db052d4d399; no voters: I0507 06:32:01.448963 7650 leader_election.cc:296] T 00000000000000000000000000000000 P 5fbef3ba10d6481bac975db052d4d399 [CANDIDATE]: Term 1 election: Requested vote from peers I0507 06:32:01.449208 7653 raft_consensus.cc:2640] T 00000000000000000000000000000000 P 5fbef3ba10d6481bac975db052d4d399 [term 1 FOLLOWER]: Leader election won for term 1 I0507 06:32:01.449859 7653 raft_consensus.cc:667] T 00000000000000000000000000000000 P 5fbef3ba10d6481bac975db052d4d399 [term 1 LEADER]: Becoming Leader. State: Replica: 5fbef3ba10d6481bac975db052d4d399, State: Running, Role: LEADER I0507 06:32:01.450338 7653 consensus_queue.cc:206] T 00000000000000000000000000000000 P 5fbef3ba10d6481bac975db052d4d399 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fbef3ba10d6481bac975db052d4d399" member_type: VOTER } I0507 06:32:01.453125 7650 sys_catalog.cc:337] T 00000000000000000000000000000000 P 5fbef3ba10d6481bac975db052d4d399 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 1 leader_uuid: "5fbef3ba10d6481bac975db052d4d399" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fbef3ba10d6481bac975db052d4d399" member_type: VOTER } } I0507 06:32:01.453101 7654 sys_catalog.cc:337] T 00000000000000000000000000000000 P 5fbef3ba10d6481bac975db052d4d399 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5fbef3ba10d6481bac975db052d4d399. Latest consensus state: current_term: 1 leader_uuid: "5fbef3ba10d6481bac975db052d4d399" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fbef3ba10d6481bac975db052d4d399" member_type: VOTER } } I0507 06:32:01.453526 7650 sys_catalog.cc:340] T 00000000000000000000000000000000 P 5fbef3ba10d6481bac975db052d4d399 [sys.catalog]: This master's current role is: LEADER I0507 06:32:01.453546 7654 sys_catalog.cc:340] T 00000000000000000000000000000000 P 5fbef3ba10d6481bac975db052d4d399 [sys.catalog]: This master's current role is: LEADER I0507 06:32:01.454473 7655 sys_catalog.cc:337] T 00000000000000000000000000000000 P 5fbef3ba10d6481bac975db052d4d399 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5fbef3ba10d6481bac975db052d4d399" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fbef3ba10d6481bac975db052d4d399" member_type: VOTER } } I0507 06:32:01.454743 7655 sys_catalog.cc:340] T 00000000000000000000000000000000 P 5fbef3ba10d6481bac975db052d4d399 [sys.catalog]: This master's current role is: LEADER I0507 06:32:01.456140 7650 sys_catalog.cc:424] T 00000000000000000000000000000000 P 5fbef3ba10d6481bac975db052d4d399 [sys.catalog]: configured and running, proceeding with master startup. I0507 06:32:01.456291 7659 catalog_manager.cc:1055] Loading table and tablet metadata into memory... I0507 06:32:01.456594 7660 catalog_manager.cc:538] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting I0507 06:32:01.460000 7659 catalog_manager.cc:1066] Initializing Kudu internal certificate authority... I0507 06:32:01.482347 7659 catalog_manager.cc:950] Generated new certificate authority record I0507 06:32:01.483623 7659 catalog_manager.cc:1075] Loading token signing keys... I0507 06:32:01.495865 7659 catalog_manager.cc:4292] T 00000000000000000000000000000000 P 5fbef3ba10d6481bac975db052d4d399: Generated new TSK 0 WARNING: Logging before InitGoogleLogging() is written to STDERR W0507 06:32:01.595309 7651 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0507 06:32:01.595557 7651 flags.cc:404] Enabled unsafe flag: --never_fsync=true W0507 06:32:01.597805 7651 flags.cc:404] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true W0507 06:32:01.597867 7651 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W0507 06:32:01.597967 7651 flags.cc:404] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.57.193 I0507 06:32:01.620725 7651 tablet_server_main.cc:78] Tablet server non-default flags: --consensus_rpc_timeout_ms=5000 --follower_unavailable_considered_failed_sec=2 --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.RestartClusterWithNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.RestartClusterWithNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-0/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.0.57.193:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.RestartClusterWithNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data/info.pb --webserver_interface=127.0.57.193 --webserver_port=0 --tablet_copy_download_file_inject_latency_ms=10000 --tserver_master_addrs=127.0.57.254:34231 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.0.57.193 --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.RestartClusterWithNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-0/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.10.0-cdh6.x-SNAPSHOT revision 7d0ee26697926e87af9ef395008799577f6ab43f build type FASTDEBUG built by jenkins at 06 May 2019 22:44:21 PST on quasar-knmpfx-1.vpc.cloudera.com build id 322017 ASAN enabled I0507 06:32:01.629930 7651 system_ntp.cc:202] NTP initialized. Skew: 500ppm Current error: 135016us I0507 06:32:01.630724 7651 fs_manager.cc:263] Metadata directory not provided I0507 06:32:01.630802 7651 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0507 06:32:01.631157 7651 server_base.cc:433] Could not load existing FS layout: Not found: could not find a healthy instance file I0507 06:32:01.631232 7651 server_base.cc:434] Attempting to create new FS layout instead I0507 06:32:01.633947 7651 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.RestartClusterWithNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data/instance: uuid: "3ce79ee2bb1348d6a36d8ce39df7d3aa" format_stamp: "Formatted at 2019-05-07 06:32:01 on dist-test-slave-dist-test-slave-rqp6" I0507 06:32:01.634544 7651 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.RestartClusterWithNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-0/wal/instance: uuid: "3ce79ee2bb1348d6a36d8ce39df7d3aa" format_stamp: "Formatted at 2019-05-07 06:32:01 on dist-test-slave-dist-test-slave-rqp6" I0507 06:32:01.641959 7651 fs_manager.cc:503] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s I0507 06:32:01.643138 7651 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0507 06:32:01.643798 7651 file_cache.cc:466] Constructed file cache lbm with capacity 419430 I0507 06:32:01.650622 7651 fs_manager.cc:419] Time spent opening block manager: real 0.006s user 0.000s sys 0.004s I0507 06:32:01.650717 7651 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.RestartClusterWithNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.RestartClusterWithNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-0/wal uuid: "3ce79ee2bb1348d6a36d8ce39df7d3aa" format_stamp: "Formatted at 2019-05-07 06:32:01 on dist-test-slave-dist-test-slave-rqp6" I0507 06:32:01.650925 7651 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.RestartClusterWithNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-0/wal metadata directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.RestartClusterWithNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-0/wal 1 data directories: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.RestartClusterWithNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0507 06:32:01.711453 7651 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0507 06:32:01.713440 7651 ts_tablet_manager.cc:356] Loaded tablet metadata (0 total tablets, 0 live tablets) I0507 06:32:01.713521 7651 ts_tablet_manager.cc:376] Registered 0 tablets I0507 06:32:01.758285 7651 rpc_server.cc:205] RPC server started. Bound to: 127.0.57.193:44481 I0507 06:32:01.759604 7651 webserver.cc:308] Webserver started at http://127.0.57.193:44839/ using document root and password file I0507 06:32:01.760397 7651 server_base.cc:608] Dumped server information to /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.RestartClusterWithNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data/info.pb I0507 06:32:01.760922 231 external_mini_cluster.cc:995] Started /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver as pid 7651 I0507 06:32:01.761600 231 external_mini_cluster.cc:933] Running /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.RestartClusterWithNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-1/wal --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.RestartClusterWithNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-1/data --block_manager=log --rpc_bind_addresses=127.0.57.194:0 --local_ip_for_outbound_sockets=127.0.57.194 --webserver_interface=127.0.57.194 --webserver_port=0 --tserver_master_addrs=127.0.57.254:34231 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.RestartClusterWithNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-1/logs --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.RestartClusterWithNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-1/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=true --tablet_copy_download_file_inject_latency_ms=10000 --follower_unavailable_considered_failed_sec=2 --consensus_rpc_timeout_ms=5000 with env {} I0507 06:32:01.788339 7781 heartbeater.cc:345] Connected to a master server at 127.0.57.254:34231 I0507 06:32:01.788607 7781 heartbeater.cc:426] Registering TS with master... I0507 06:32:01.789224 7781 heartbeater.cc:475] Master 127.0.57.254:34231 requested a full tablet report, sending... I0507 06:32:01.790925 7617 ts_manager.cc:159] Registered new tserver with Master: 3ce79ee2bb1348d6a36d8ce39df7d3aa (127.0.57.193:44481) I0507 06:32:01.792484 7617 master_service.cc:273] Signed X509 certificate for tserver {username='slave'} at 127.0.57.193:58785 WARNING: Logging before InitGoogleLogging() is written to STDERR W0507 06:32:01.942695 7782 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0507 06:32:01.942945 7782 flags.cc:404] Enabled unsafe flag: --never_fsync=true W0507 06:32:01.945158 7782 flags.cc:404] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true W0507 06:32:01.945219 7782 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W0507 06:32:01.945322 7782 flags.cc:404] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.57.194 I0507 06:32:01.949470 7782 tablet_server_main.cc:78] Tablet server non-default flags: --consensus_rpc_timeout_ms=5000 --follower_unavailable_considered_failed_sec=2 --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.RestartClusterWithNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-1/data --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.RestartClusterWithNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-1/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.0.57.194:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.RestartClusterWithNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-1/data/info.pb --webserver_interface=127.0.57.194 --webserver_port=0 --tablet_copy_download_file_inject_latency_ms=10000 --tserver_master_addrs=127.0.57.254:34231 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.0.57.194 --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.RestartClusterWithNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-1/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.10.0-cdh6.x-SNAPSHOT revision 7d0ee26697926e87af9ef395008799577f6ab43f build type FASTDEBUG built by jenkins at 06 May 2019 22:44:21 PST on quasar-knmpfx-1.vpc.cloudera.com build id 322017 ASAN enabled I0507 06:32:01.958729 7782 system_ntp.cc:202] NTP initialized. Skew: 500ppm Current error: 135016us I0507 06:32:01.959535 7782 fs_manager.cc:263] Metadata directory not provided I0507 06:32:01.959612 7782 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0507 06:32:01.959946 7782 server_base.cc:433] Could not load existing FS layout: Not found: could not find a healthy instance file I0507 06:32:01.960031 7782 server_base.cc:434] Attempting to create new FS layout instead I0507 06:32:01.962810 7782 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.RestartClusterWithNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-1/data/instance: uuid: "d9fd45507a8e435abc710b00a27538c6" format_stamp: "Formatted at 2019-05-07 06:32:01 on dist-test-slave-dist-test-slave-rqp6" I0507 06:32:01.963444 7782 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.RestartClusterWithNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-1/wal/instance: uuid: "d9fd45507a8e435abc710b00a27538c6" format_stamp: "Formatted at 2019-05-07 06:32:01 on dist-test-slave-dist-test-slave-rqp6" I0507 06:32:01.970834 7782 fs_manager.cc:503] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s I0507 06:32:01.972070 7782 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0507 06:32:01.972762 7782 file_cache.cc:466] Constructed file cache lbm with capacity 419430 I0507 06:32:01.976153 7782 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s I0507 06:32:01.976253 7782 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.RestartClusterWithNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.RestartClusterWithNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-1/wal uuid: "d9fd45507a8e435abc710b00a27538c6" format_stamp: "Formatted at 2019-05-07 06:32:01 on dist-test-slave-dist-test-slave-rqp6" I0507 06:32:01.976423 7782 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.RestartClusterWithNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-1/wal metadata directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.RestartClusterWithNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-1/wal 1 data directories: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.RestartClusterWithNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-1/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0507 06:32:02.012377 7782 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0507 06:32:02.014850 7782 ts_tablet_manager.cc:356] Loaded tablet metadata (0 total tablets, 0 live tablets) I0507 06:32:02.014977 7782 ts_tablet_manager.cc:376] Registered 0 tablets I0507 06:32:02.044620 7782 rpc_server.cc:205] RPC server started. Bound to: 127.0.57.194:44183 I0507 06:32:02.046008 7782 webserver.cc:308] Webserver started at http://127.0.57.194:35977/ using document root and password file I0507 06:32:02.046880 7782 server_base.cc:608] Dumped server information to /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.RestartClusterWithNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-1/data/info.pb I0507 06:32:02.049018 231 external_mini_cluster.cc:995] Started /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver as pid 7782 I0507 06:32:02.049713 231 external_mini_cluster.cc:933] Running /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.RestartClusterWithNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-2/wal --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.RestartClusterWithNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-2/data --block_manager=log --rpc_bind_addresses=127.0.57.195:0 --local_ip_for_outbound_sockets=127.0.57.195 --webserver_interface=127.0.57.195 --webserver_port=0 --tserver_master_addrs=127.0.57.254:34231 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.RestartClusterWithNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-2/logs --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.RestartClusterWithNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-2/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=true --tablet_copy_download_file_inject_latency_ms=10000 --follower_unavailable_considered_failed_sec=2 --consensus_rpc_timeout_ms=5000 with env {} I0507 06:32:02.063115 7904 heartbeater.cc:345] Connected to a master server at 127.0.57.254:34231 I0507 06:32:02.063360 7904 heartbeater.cc:426] Registering TS with master... I0507 06:32:02.063971 7904 heartbeater.cc:475] Master 127.0.57.254:34231 requested a full tablet report, sending... I0507 06:32:02.065331 7617 ts_manager.cc:159] Registered new tserver with Master: d9fd45507a8e435abc710b00a27538c6 (127.0.57.194:44183) I0507 06:32:02.066363 7617 master_service.cc:273] Signed X509 certificate for tserver {username='slave'} at 127.0.57.194:49955 WARNING: Logging before InitGoogleLogging() is written to STDERR W0507 06:32:02.213578 7907 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0507 06:32:02.213822 7907 flags.cc:404] Enabled unsafe flag: --never_fsync=true W0507 06:32:02.215884 7907 flags.cc:404] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true W0507 06:32:02.215941 7907 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W0507 06:32:02.216035 7907 flags.cc:404] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.57.195 I0507 06:32:02.220213 7907 tablet_server_main.cc:78] Tablet server non-default flags: --consensus_rpc_timeout_ms=5000 --follower_unavailable_considered_failed_sec=2 --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.RestartClusterWithNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-2/data --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.RestartClusterWithNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-2/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.0.57.195:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.RestartClusterWithNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-2/data/info.pb --webserver_interface=127.0.57.195 --webserver_port=0 --tablet_copy_download_file_inject_latency_ms=10000 --tserver_master_addrs=127.0.57.254:34231 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.0.57.195 --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.RestartClusterWithNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-2/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.10.0-cdh6.x-SNAPSHOT revision 7d0ee26697926e87af9ef395008799577f6ab43f build type FASTDEBUG built by jenkins at 06 May 2019 22:44:21 PST on quasar-knmpfx-1.vpc.cloudera.com build id 322017 ASAN enabled I0507 06:32:02.227555 7907 system_ntp.cc:202] NTP initialized. Skew: 500ppm Current error: 135516us I0507 06:32:02.228209 7907 fs_manager.cc:263] Metadata directory not provided I0507 06:32:02.228265 7907 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0507 06:32:02.228504 7907 server_base.cc:433] Could not load existing FS layout: Not found: could not find a healthy instance file I0507 06:32:02.228585 7907 server_base.cc:434] Attempting to create new FS layout instead I0507 06:32:02.230903 7907 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.RestartClusterWithNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-2/data/instance: uuid: "2781869a927549908dde07bc5019d0e4" format_stamp: "Formatted at 2019-05-07 06:32:02 on dist-test-slave-dist-test-slave-rqp6" I0507 06:32:02.231398 7907 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.RestartClusterWithNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-2/wal/instance: uuid: "2781869a927549908dde07bc5019d0e4" format_stamp: "Formatted at 2019-05-07 06:32:02 on dist-test-slave-dist-test-slave-rqp6" I0507 06:32:02.238123 7907 fs_manager.cc:503] Time spent creating directory manager: real 0.006s user 0.003s sys 0.005s I0507 06:32:02.239147 7907 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0507 06:32:02.239673 7907 file_cache.cc:466] Constructed file cache lbm with capacity 419430 I0507 06:32:02.242923 7907 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s I0507 06:32:02.243010 7907 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.RestartClusterWithNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.RestartClusterWithNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-2/wal uuid: "2781869a927549908dde07bc5019d0e4" format_stamp: "Formatted at 2019-05-07 06:32:02 on dist-test-slave-dist-test-slave-rqp6" I0507 06:32:02.243149 7907 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.RestartClusterWithNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-2/wal metadata directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.RestartClusterWithNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-2/wal 1 data directories: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.RestartClusterWithNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-2/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0507 06:32:02.323217 7907 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0507 06:32:02.326139 7907 ts_tablet_manager.cc:356] Loaded tablet metadata (0 total tablets, 0 live tablets) I0507 06:32:02.326226 7907 ts_tablet_manager.cc:376] Registered 0 tablets I0507 06:32:02.351828 7907 rpc_server.cc:205] RPC server started. Bound to: 127.0.57.195:40501 I0507 06:32:02.353085 7907 webserver.cc:308] Webserver started at http://127.0.57.195:42821/ using document root and password file I0507 06:32:02.353862 7907 server_base.cc:608] Dumped server information to /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.RestartClusterWithNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-2/data/info.pb I0507 06:32:02.361102 231 external_mini_cluster.cc:995] Started /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver as pid 7907 I0507 06:32:02.361757 231 external_mini_cluster.cc:933] Running /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.RestartClusterWithNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-3/wal --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.RestartClusterWithNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-3/data --block_manager=log --rpc_bind_addresses=127.0.57.196:0 --local_ip_for_outbound_sockets=127.0.57.196 --webserver_interface=127.0.57.196 --webserver_port=0 --tserver_master_addrs=127.0.57.254:34231 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.RestartClusterWithNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-3/logs --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.RestartClusterWithNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-3/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=true --tablet_copy_download_file_inject_latency_ms=10000 --follower_unavailable_considered_failed_sec=2 --consensus_rpc_timeout_ms=5000 with env {} I0507 06:32:02.367224 8026 heartbeater.cc:345] Connected to a master server at 127.0.57.254:34231 I0507 06:32:02.367442 8026 heartbeater.cc:426] Registering TS with master... I0507 06:32:02.367988 8026 heartbeater.cc:475] Master 127.0.57.254:34231 requested a full tablet report, sending... I0507 06:32:02.369184 7617 ts_manager.cc:159] Registered new tserver with Master: 2781869a927549908dde07bc5019d0e4 (127.0.57.195:40501) I0507 06:32:02.370332 7617 master_service.cc:273] Signed X509 certificate for tserver {username='slave'} at 127.0.57.195:36421 WARNING: Logging before InitGoogleLogging() is written to STDERR W0507 06:32:02.526193 8029 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0507 06:32:02.526430 8029 flags.cc:404] Enabled unsafe flag: --never_fsync=true W0507 06:32:02.528544 8029 flags.cc:404] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true W0507 06:32:02.528605 8029 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W0507 06:32:02.528722 8029 flags.cc:404] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.57.196 I0507 06:32:02.532796 8029 tablet_server_main.cc:78] Tablet server non-default flags: --consensus_rpc_timeout_ms=5000 --follower_unavailable_considered_failed_sec=2 --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.RestartClusterWithNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-3/data --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.RestartClusterWithNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-3/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.0.57.196:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.RestartClusterWithNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-3/data/info.pb --webserver_interface=127.0.57.196 --webserver_port=0 --tablet_copy_download_file_inject_latency_ms=10000 --tserver_master_addrs=127.0.57.254:34231 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.0.57.196 --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.RestartClusterWithNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-3/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.10.0-cdh6.x-SNAPSHOT revision 7d0ee26697926e87af9ef395008799577f6ab43f build type FASTDEBUG built by jenkins at 06 May 2019 22:44:21 PST on quasar-knmpfx-1.vpc.cloudera.com build id 322017 ASAN enabled I0507 06:32:02.540931 8029 system_ntp.cc:202] NTP initialized. Skew: 500ppm Current error: 135516us I0507 06:32:02.541666 8029 fs_manager.cc:263] Metadata directory not provided I0507 06:32:02.541745 8029 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0507 06:32:02.542026 8029 server_base.cc:433] Could not load existing FS layout: Not found: could not find a healthy instance file I0507 06:32:02.542096 8029 server_base.cc:434] Attempting to create new FS layout instead I0507 06:32:02.544713 8029 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.RestartClusterWithNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-3/data/instance: uuid: "b5c46129f1994ab28f3710e8e392ceab" format_stamp: "Formatted at 2019-05-07 06:32:02 on dist-test-slave-dist-test-slave-rqp6" I0507 06:32:02.545285 8029 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.RestartClusterWithNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-3/wal/instance: uuid: "b5c46129f1994ab28f3710e8e392ceab" format_stamp: "Formatted at 2019-05-07 06:32:02 on dist-test-slave-dist-test-slave-rqp6" I0507 06:32:02.551967 8029 fs_manager.cc:503] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s I0507 06:32:02.553052 8029 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0507 06:32:02.553705 8029 file_cache.cc:466] Constructed file cache lbm with capacity 419430 I0507 06:32:02.556711 8029 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s I0507 06:32:02.556784 8029 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.RestartClusterWithNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.RestartClusterWithNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-3/wal uuid: "b5c46129f1994ab28f3710e8e392ceab" format_stamp: "Formatted at 2019-05-07 06:32:02 on dist-test-slave-dist-test-slave-rqp6" I0507 06:32:02.556931 8029 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.RestartClusterWithNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-3/wal metadata directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.RestartClusterWithNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-3/wal 1 data directories: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.RestartClusterWithNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-3/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0507 06:32:02.576318 8029 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0507 06:32:02.578194 8029 ts_tablet_manager.cc:356] Loaded tablet metadata (0 total tablets, 0 live tablets) I0507 06:32:02.578274 8029 ts_tablet_manager.cc:376] Registered 0 tablets I0507 06:32:02.605399 8029 rpc_server.cc:205] RPC server started. Bound to: 127.0.57.196:34535 I0507 06:32:02.606609 8029 webserver.cc:308] Webserver started at http://127.0.57.196:35825/ using document root and password file I0507 06:32:02.607278 8029 server_base.cc:608] Dumped server information to /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.RestartClusterWithNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-3/data/info.pb I0507 06:32:02.611594 231 external_mini_cluster.cc:995] Started /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver as pid 8029 I0507 06:32:02.612426 231 external_mini_cluster.cc:933] Running /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.RestartClusterWithNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-4/wal --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.RestartClusterWithNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-4/data --block_manager=log --rpc_bind_addresses=127.0.57.197:0 --local_ip_for_outbound_sockets=127.0.57.197 --webserver_interface=127.0.57.197 --webserver_port=0 --tserver_master_addrs=127.0.57.254:34231 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.RestartClusterWithNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-4/logs --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.RestartClusterWithNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-4/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=true --tablet_copy_download_file_inject_latency_ms=10000 --follower_unavailable_considered_failed_sec=2 --consensus_rpc_timeout_ms=5000 with env {} I0507 06:32:02.627146 8148 heartbeater.cc:345] Connected to a master server at 127.0.57.254:34231 I0507 06:32:02.627393 8148 heartbeater.cc:426] Registering TS with master... I0507 06:32:02.628024 8148 heartbeater.cc:475] Master 127.0.57.254:34231 requested a full tablet report, sending... I0507 06:32:02.629355 7617 ts_manager.cc:159] Registered new tserver with Master: b5c46129f1994ab28f3710e8e392ceab (127.0.57.196:34535) I0507 06:32:02.630759 7617 master_service.cc:273] Signed X509 certificate for tserver {username='slave'} at 127.0.57.196:46517 I0507 06:32:02.795086 7781 heartbeater.cc:467] Master 127.0.57.254:34231 was elected leader, sending a full tablet report... WARNING: Logging before InitGoogleLogging() is written to STDERR W0507 06:32:02.795243 8151 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0507 06:32:02.795583 8151 flags.cc:404] Enabled unsafe flag: --never_fsync=true W0507 06:32:02.798259 8151 flags.cc:404] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true W0507 06:32:02.798324 8151 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W0507 06:32:02.798439 8151 flags.cc:404] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.57.197 I0507 06:32:02.802868 8151 tablet_server_main.cc:78] Tablet server non-default flags: --consensus_rpc_timeout_ms=5000 --follower_unavailable_considered_failed_sec=2 --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.RestartClusterWithNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-4/data --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.RestartClusterWithNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-4/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.0.57.197:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.RestartClusterWithNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-4/data/info.pb --webserver_interface=127.0.57.197 --webserver_port=0 --tablet_copy_download_file_inject_latency_ms=10000 --tserver_master_addrs=127.0.57.254:34231 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.0.57.197 --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.RestartClusterWithNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-4/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.10.0-cdh6.x-SNAPSHOT revision 7d0ee26697926e87af9ef395008799577f6ab43f build type FASTDEBUG built by jenkins at 06 May 2019 22:44:21 PST on quasar-knmpfx-1.vpc.cloudera.com build id 322017 ASAN enabled I0507 06:32:02.811424 8151 system_ntp.cc:202] NTP initialized. Skew: 500ppm Current error: 135516us I0507 06:32:02.812248 8151 fs_manager.cc:263] Metadata directory not provided I0507 06:32:02.812314 8151 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0507 06:32:02.812650 8151 server_base.cc:433] Could not load existing FS layout: Not found: could not find a healthy instance file I0507 06:32:02.812737 8151 server_base.cc:434] Attempting to create new FS layout instead I0507 06:32:02.815605 8151 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.RestartClusterWithNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-4/data/instance: uuid: "bb5a49b22d4048608d9e7b762d749fb2" format_stamp: "Formatted at 2019-05-07 06:32:02 on dist-test-slave-dist-test-slave-rqp6" I0507 06:32:02.816211 8151 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.RestartClusterWithNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-4/wal/instance: uuid: "bb5a49b22d4048608d9e7b762d749fb2" format_stamp: "Formatted at 2019-05-07 06:32:02 on dist-test-slave-dist-test-slave-rqp6" I0507 06:32:02.824036 8151 fs_manager.cc:503] Time spent creating directory manager: real 0.008s user 0.009s sys 0.000s I0507 06:32:02.825297 8151 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0507 06:32:02.826050 8151 file_cache.cc:466] Constructed file cache lbm with capacity 419430 I0507 06:32:02.830072 8151 fs_manager.cc:419] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s I0507 06:32:02.830173 8151 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.RestartClusterWithNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.RestartClusterWithNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-4/wal uuid: "bb5a49b22d4048608d9e7b762d749fb2" format_stamp: "Formatted at 2019-05-07 06:32:02 on dist-test-slave-dist-test-slave-rqp6" I0507 06:32:02.830353 8151 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.RestartClusterWithNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-4/wal metadata directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.RestartClusterWithNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-4/wal 1 data directories: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.RestartClusterWithNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-4/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0507 06:32:02.864457 8151 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0507 06:32:02.866451 8151 ts_tablet_manager.cc:356] Loaded tablet metadata (0 total tablets, 0 live tablets) I0507 06:32:02.866549 8151 ts_tablet_manager.cc:376] Registered 0 tablets I0507 06:32:02.896004 8151 rpc_server.cc:205] RPC server started. Bound to: 127.0.57.197:44221 I0507 06:32:02.897292 8151 webserver.cc:308] Webserver started at http://127.0.57.197:41379/ using document root and password file I0507 06:32:02.897997 8151 server_base.cc:608] Dumped server information to /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.RestartClusterWithNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-4/data/info.pb I0507 06:32:02.906517 231 external_mini_cluster.cc:995] Started /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver as pid 8151 I0507 06:32:02.907431 231 external_mini_cluster.cc:933] Running /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.RestartClusterWithNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-5/wal --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.RestartClusterWithNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-5/data --block_manager=log --rpc_bind_addresses=127.0.57.198:0 --local_ip_for_outbound_sockets=127.0.57.198 --webserver_interface=127.0.57.198 --webserver_port=0 --tserver_master_addrs=127.0.57.254:34231 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.RestartClusterWithNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-5/logs --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.RestartClusterWithNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-5/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=true --tablet_copy_download_file_inject_latency_ms=10000 --follower_unavailable_considered_failed_sec=2 --consensus_rpc_timeout_ms=5000 with env {} I0507 06:32:02.912756 8270 heartbeater.cc:345] Connected to a master server at 127.0.57.254:34231 I0507 06:32:02.913022 8270 heartbeater.cc:426] Registering TS with master... I0507 06:32:02.913738 8270 heartbeater.cc:475] Master 127.0.57.254:34231 requested a full tablet report, sending... I0507 06:32:02.915076 7617 ts_manager.cc:159] Registered new tserver with Master: bb5a49b22d4048608d9e7b762d749fb2 (127.0.57.197:44221) I0507 06:32:02.916050 7617 master_service.cc:273] Signed X509 certificate for tserver {username='slave'} at 127.0.57.197:57777 I0507 06:32:03.068682 7904 heartbeater.cc:467] Master 127.0.57.254:34231 was elected leader, sending a full tablet report... WARNING: Logging before InitGoogleLogging() is written to STDERR W0507 06:32:03.091449 8273 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0507 06:32:03.091722 8273 flags.cc:404] Enabled unsafe flag: --never_fsync=true W0507 06:32:03.094144 8273 flags.cc:404] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true W0507 06:32:03.094254 8273 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W0507 06:32:03.094388 8273 flags.cc:404] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.57.198 I0507 06:32:03.099069 8273 tablet_server_main.cc:78] Tablet server non-default flags: --consensus_rpc_timeout_ms=5000 --follower_unavailable_considered_failed_sec=2 --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.RestartClusterWithNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-5/data --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.RestartClusterWithNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-5/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.0.57.198:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.RestartClusterWithNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-5/data/info.pb --webserver_interface=127.0.57.198 --webserver_port=0 --tablet_copy_download_file_inject_latency_ms=10000 --tserver_master_addrs=127.0.57.254:34231 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.0.57.198 --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.RestartClusterWithNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-5/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.10.0-cdh6.x-SNAPSHOT revision 7d0ee26697926e87af9ef395008799577f6ab43f build type FASTDEBUG built by jenkins at 06 May 2019 22:44:21 PST on quasar-knmpfx-1.vpc.cloudera.com build id 322017 ASAN enabled I0507 06:32:03.108129 8273 system_ntp.cc:202] NTP initialized. Skew: 500ppm Current error: 136016us I0507 06:32:03.108899 8273 fs_manager.cc:263] Metadata directory not provided I0507 06:32:03.108968 8273 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0507 06:32:03.109371 8273 server_base.cc:433] Could not load existing FS layout: Not found: could not find a healthy instance file I0507 06:32:03.109441 8273 server_base.cc:434] Attempting to create new FS layout instead I0507 06:32:03.112221 8273 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.RestartClusterWithNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-5/data/instance: uuid: "3e033fe6ec6f4fe595bdcd8a95a9f4e8" format_stamp: "Formatted at 2019-05-07 06:32:03 on dist-test-slave-dist-test-slave-rqp6" I0507 06:32:03.112852 8273 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.RestartClusterWithNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-5/wal/instance: uuid: "3e033fe6ec6f4fe595bdcd8a95a9f4e8" format_stamp: "Formatted at 2019-05-07 06:32:03 on dist-test-slave-dist-test-slave-rqp6" I0507 06:32:03.120885 8273 fs_manager.cc:503] Time spent creating directory manager: real 0.008s user 0.006s sys 0.000s I0507 06:32:03.122189 8273 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0507 06:32:03.122812 8273 file_cache.cc:466] Constructed file cache lbm with capacity 419430 I0507 06:32:03.126401 8273 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s I0507 06:32:03.126508 8273 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.RestartClusterWithNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.RestartClusterWithNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-5/wal uuid: "3e033fe6ec6f4fe595bdcd8a95a9f4e8" format_stamp: "Formatted at 2019-05-07 06:32:03 on dist-test-slave-dist-test-slave-rqp6" I0507 06:32:03.126729 8273 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.RestartClusterWithNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-5/wal metadata directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.RestartClusterWithNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-5/wal 1 data directories: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.RestartClusterWithNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-5/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0507 06:32:03.154245 8273 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0507 06:32:03.156219 8273 ts_tablet_manager.cc:356] Loaded tablet metadata (0 total tablets, 0 live tablets) I0507 06:32:03.156395 8273 ts_tablet_manager.cc:376] Registered 0 tablets I0507 06:32:03.185925 8273 rpc_server.cc:205] RPC server started. Bound to: 127.0.57.198:42585 I0507 06:32:03.187335 8273 webserver.cc:308] Webserver started at http://127.0.57.198:34145/ using document root and password file I0507 06:32:03.188107 8273 server_base.cc:608] Dumped server information to /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.RestartClusterWithNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-5/data/info.pb I0507 06:32:03.197841 231 external_mini_cluster.cc:995] Started /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver as pid 8273 I0507 06:32:03.198562 231 external_mini_cluster.cc:933] Running /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.RestartClusterWithNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-6/wal --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.RestartClusterWithNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-6/data --block_manager=log --rpc_bind_addresses=127.0.57.199:0 --local_ip_for_outbound_sockets=127.0.57.199 --webserver_interface=127.0.57.199 --webserver_port=0 --tserver_master_addrs=127.0.57.254:34231 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.RestartClusterWithNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-6/logs --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.RestartClusterWithNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-6/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=true --tablet_copy_download_file_inject_latency_ms=10000 --follower_unavailable_considered_failed_sec=2 --consensus_rpc_timeout_ms=5000 with env {} I0507 06:32:03.201437 8392 heartbeater.cc:345] Connected to a master server at 127.0.57.254:34231 I0507 06:32:03.201712 8392 heartbeater.cc:426] Registering TS with master... I0507 06:32:03.202392 8392 heartbeater.cc:475] Master 127.0.57.254:34231 requested a full tablet report, sending... I0507 06:32:03.203661 7617 ts_manager.cc:159] Registered new tserver with Master: 3e033fe6ec6f4fe595bdcd8a95a9f4e8 (127.0.57.198:42585) I0507 06:32:03.204712 7617 master_service.cc:273] Signed X509 certificate for tserver {username='slave'} at 127.0.57.198:40531 WARNING: Logging before InitGoogleLogging() is written to STDERR W0507 06:32:03.369819 8395 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0507 06:32:03.370059 8395 flags.cc:404] Enabled unsafe flag: --never_fsync=true W0507 06:32:03.372191 8395 flags.cc:404] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true W0507 06:32:03.372243 8395 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W0507 06:32:03.372341 8395 flags.cc:404] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.57.199 I0507 06:32:03.372742 8026 heartbeater.cc:467] Master 127.0.57.254:34231 was elected leader, sending a full tablet report... I0507 06:32:03.377266 8395 tablet_server_main.cc:78] Tablet server non-default flags: --consensus_rpc_timeout_ms=5000 --follower_unavailable_considered_failed_sec=2 --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.RestartClusterWithNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-6/data --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.RestartClusterWithNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-6/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.0.57.199:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.RestartClusterWithNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-6/data/info.pb --webserver_interface=127.0.57.199 --webserver_port=0 --tablet_copy_download_file_inject_latency_ms=10000 --tserver_master_addrs=127.0.57.254:34231 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.0.57.199 --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.RestartClusterWithNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-6/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.10.0-cdh6.x-SNAPSHOT revision 7d0ee26697926e87af9ef395008799577f6ab43f build type FASTDEBUG built by jenkins at 06 May 2019 22:44:21 PST on quasar-knmpfx-1.vpc.cloudera.com build id 322017 ASAN enabled I0507 06:32:03.385551 8395 system_ntp.cc:202] NTP initialized. Skew: 500ppm Current error: 136016us I0507 06:32:03.386303 8395 fs_manager.cc:263] Metadata directory not provided I0507 06:32:03.386365 8395 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0507 06:32:03.386667 8395 server_base.cc:433] Could not load existing FS layout: Not found: could not find a healthy instance file I0507 06:32:03.386732 8395 server_base.cc:434] Attempting to create new FS layout instead I0507 06:32:03.389295 8395 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.RestartClusterWithNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-6/data/instance: uuid: "44a1757d4e604357a1991e808f009e56" format_stamp: "Formatted at 2019-05-07 06:32:03 on dist-test-slave-dist-test-slave-rqp6" I0507 06:32:03.389876 8395 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.RestartClusterWithNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-6/wal/instance: uuid: "44a1757d4e604357a1991e808f009e56" format_stamp: "Formatted at 2019-05-07 06:32:03 on dist-test-slave-dist-test-slave-rqp6" I0507 06:32:03.397660 8395 fs_manager.cc:503] Time spent creating directory manager: real 0.008s user 0.006s sys 0.001s I0507 06:32:03.398970 8395 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0507 06:32:03.399646 8395 file_cache.cc:466] Constructed file cache lbm with capacity 419430 I0507 06:32:03.403129 8395 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s I0507 06:32:03.403223 8395 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.RestartClusterWithNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.RestartClusterWithNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-6/wal uuid: "44a1757d4e604357a1991e808f009e56" format_stamp: "Formatted at 2019-05-07 06:32:03 on dist-test-slave-dist-test-slave-rqp6" I0507 06:32:03.403419 8395 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.RestartClusterWithNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-6/wal metadata directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.RestartClusterWithNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-6/wal 1 data directories: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.RestartClusterWithNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-6/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0507 06:32:03.439852 8395 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0507 06:32:03.441536 8395 ts_tablet_manager.cc:356] Loaded tablet metadata (0 total tablets, 0 live tablets) I0507 06:32:03.441628 8395 ts_tablet_manager.cc:376] Registered 0 tablets I0507 06:32:03.468252 8395 rpc_server.cc:205] RPC server started. Bound to: 127.0.57.199:44671 I0507 06:32:03.469578 8395 webserver.cc:308] Webserver started at http://127.0.57.199:42325/ using document root and password file I0507 06:32:03.470336 8395 server_base.cc:608] Dumped server information to /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.RestartClusterWithNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-6/data/info.pb I0507 06:32:03.478328 231 external_mini_cluster.cc:995] Started /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver as pid 8395 I0507 06:32:03.485287 8514 heartbeater.cc:345] Connected to a master server at 127.0.57.254:34231 I0507 06:32:03.485517 8514 heartbeater.cc:426] Registering TS with master... I0507 06:32:03.486096 8514 heartbeater.cc:475] Master 127.0.57.254:34231 requested a full tablet report, sending... I0507 06:32:03.487365 7617 ts_manager.cc:159] Registered new tserver with Master: 44a1757d4e604357a1991e808f009e56 (127.0.57.199:44671) I0507 06:32:03.488345 7617 master_service.cc:273] Signed X509 certificate for tserver {username='slave'} at 127.0.57.199:52945 I0507 06:32:03.489905 231 external_mini_cluster.cc:623] 7 TS(s) registered with all masters I0507 06:32:03.502184 7617 catalog_manager.cc:1415] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38312: name: "TestTable" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 3 split_rows_range_bounds { } partition_schema { range_schema { columns { name: "key" } } } I0507 06:32:03.534874 7718 tablet_service.cc:1016] Processing CreateTablet for tablet 3a7d8f424df742599d3f6d48d9b1a37d (table=TestTable [id=4245f699bfa84b548c6ed127e9d69518]), partition=RANGE (key) PARTITION UNBOUNDED I0507 06:32:03.535619 7963 tablet_service.cc:1016] Processing CreateTablet for tablet 3a7d8f424df742599d3f6d48d9b1a37d (table=TestTable [id=4245f699bfa84b548c6ed127e9d69518]), partition=RANGE (key) PARTITION UNBOUNDED I0507 06:32:03.535511 7841 tablet_service.cc:1016] Processing CreateTablet for tablet 3a7d8f424df742599d3f6d48d9b1a37d (table=TestTable [id=4245f699bfa84b548c6ed127e9d69518]), partition=RANGE (key) PARTITION UNBOUNDED I0507 06:32:03.535878 7718 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet 3a7d8f424df742599d3f6d48d9b1a37d. 1 dirs total, 0 dirs full, 0 dirs failed I0507 06:32:03.536316 7963 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet 3a7d8f424df742599d3f6d48d9b1a37d. 1 dirs total, 0 dirs full, 0 dirs failed I0507 06:32:03.536388 7841 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet 3a7d8f424df742599d3f6d48d9b1a37d. 1 dirs total, 0 dirs full, 0 dirs failed I0507 06:32:03.545971 8534 tablet_bootstrap.cc:439] T 3a7d8f424df742599d3f6d48d9b1a37d P d9fd45507a8e435abc710b00a27538c6: Bootstrap starting. I0507 06:32:03.546610 8535 tablet_bootstrap.cc:439] T 3a7d8f424df742599d3f6d48d9b1a37d P 2781869a927549908dde07bc5019d0e4: Bootstrap starting. I0507 06:32:03.549060 8536 tablet_bootstrap.cc:439] T 3a7d8f424df742599d3f6d48d9b1a37d P 3ce79ee2bb1348d6a36d8ce39df7d3aa: Bootstrap starting. I0507 06:32:03.551306 8536 tablet_bootstrap.cc:590] T 3a7d8f424df742599d3f6d48d9b1a37d P 3ce79ee2bb1348d6a36d8ce39df7d3aa: No blocks or log segments found. Creating new log. I0507 06:32:03.553500 8534 tablet_bootstrap.cc:590] T 3a7d8f424df742599d3f6d48d9b1a37d P d9fd45507a8e435abc710b00a27538c6: No blocks or log segments found. Creating new log. I0507 06:32:03.554522 8534 log.cc:525] T 3a7d8f424df742599d3f6d48d9b1a37d P d9fd45507a8e435abc710b00a27538c6: Log is configured to *not* fsync() on all Append() calls I0507 06:32:03.556016 8535 tablet_bootstrap.cc:590] T 3a7d8f424df742599d3f6d48d9b1a37d P 2781869a927549908dde07bc5019d0e4: No blocks or log segments found. Creating new log. I0507 06:32:03.557435 8536 log.cc:525] T 3a7d8f424df742599d3f6d48d9b1a37d P 3ce79ee2bb1348d6a36d8ce39df7d3aa: Log is configured to *not* fsync() on all Append() calls I0507 06:32:03.558085 8535 log.cc:525] T 3a7d8f424df742599d3f6d48d9b1a37d P 2781869a927549908dde07bc5019d0e4: Log is configured to *not* fsync() on all Append() calls I0507 06:32:03.559852 8534 tablet_bootstrap.cc:439] T 3a7d8f424df742599d3f6d48d9b1a37d P d9fd45507a8e435abc710b00a27538c6: No bootstrap required, opened a new log I0507 06:32:03.559948 8536 tablet_bootstrap.cc:439] T 3a7d8f424df742599d3f6d48d9b1a37d P 3ce79ee2bb1348d6a36d8ce39df7d3aa: No bootstrap required, opened a new log I0507 06:32:03.560132 8534 ts_tablet_manager.cc:1053] T 3a7d8f424df742599d3f6d48d9b1a37d P d9fd45507a8e435abc710b00a27538c6: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.000s I0507 06:32:03.560212 8536 ts_tablet_manager.cc:1053] T 3a7d8f424df742599d3f6d48d9b1a37d P 3ce79ee2bb1348d6a36d8ce39df7d3aa: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.000s I0507 06:32:03.565783 8535 tablet_bootstrap.cc:439] T 3a7d8f424df742599d3f6d48d9b1a37d P 2781869a927549908dde07bc5019d0e4: No bootstrap required, opened a new log I0507 06:32:03.565542 8534 raft_consensus.cc:340] T 3a7d8f424df742599d3f6d48d9b1a37d P d9fd45507a8e435abc710b00a27538c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ce79ee2bb1348d6a36d8ce39df7d3aa" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 44481 } } peers { permanent_uuid: "d9fd45507a8e435abc710b00a27538c6" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 44183 } } peers { permanent_uuid: "2781869a927549908dde07bc5019d0e4" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 40501 } } I0507 06:32:03.565888 8534 raft_consensus.cc:366] T 3a7d8f424df742599d3f6d48d9b1a37d P d9fd45507a8e435abc710b00a27538c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0507 06:32:03.565954 8534 raft_consensus.cc:705] T 3a7d8f424df742599d3f6d48d9b1a37d P d9fd45507a8e435abc710b00a27538c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9fd45507a8e435abc710b00a27538c6, State: Initialized, Role: FOLLOWER I0507 06:32:03.566056 8535 ts_tablet_manager.cc:1053] T 3a7d8f424df742599d3f6d48d9b1a37d P 2781869a927549908dde07bc5019d0e4: Time spent bootstrapping tablet: real 0.020s user 0.002s sys 0.002s I0507 06:32:03.566045 8536 raft_consensus.cc:340] T 3a7d8f424df742599d3f6d48d9b1a37d P 3ce79ee2bb1348d6a36d8ce39df7d3aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ce79ee2bb1348d6a36d8ce39df7d3aa" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 44481 } } peers { permanent_uuid: "d9fd45507a8e435abc710b00a27538c6" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 44183 } } peers { permanent_uuid: "2781869a927549908dde07bc5019d0e4" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 40501 } } I0507 06:32:03.566408 8536 raft_consensus.cc:366] T 3a7d8f424df742599d3f6d48d9b1a37d P 3ce79ee2bb1348d6a36d8ce39df7d3aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0507 06:32:03.566463 8536 raft_consensus.cc:705] T 3a7d8f424df742599d3f6d48d9b1a37d P 3ce79ee2bb1348d6a36d8ce39df7d3aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ce79ee2bb1348d6a36d8ce39df7d3aa, State: Initialized, Role: FOLLOWER I0507 06:32:03.566426 8534 consensus_queue.cc:229] T 3a7d8f424df742599d3f6d48d9b1a37d P d9fd45507a8e435abc710b00a27538c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ce79ee2bb1348d6a36d8ce39df7d3aa" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 44481 } } peers { permanent_uuid: "d9fd45507a8e435abc710b00a27538c6" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 44183 } } peers { permanent_uuid: "2781869a927549908dde07bc5019d0e4" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 40501 } } I0507 06:32:03.566869 8536 consensus_queue.cc:229] T 3a7d8f424df742599d3f6d48d9b1a37d P 3ce79ee2bb1348d6a36d8ce39df7d3aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ce79ee2bb1348d6a36d8ce39df7d3aa" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 44481 } } peers { permanent_uuid: "d9fd45507a8e435abc710b00a27538c6" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 44183 } } peers { permanent_uuid: "2781869a927549908dde07bc5019d0e4" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 40501 } } I0507 06:32:03.568928 8534 ts_tablet_manager.cc:1081] T 3a7d8f424df742599d3f6d48d9b1a37d P d9fd45507a8e435abc710b00a27538c6: Time spent starting tablet: real 0.009s user 0.004s sys 0.004s I0507 06:32:03.569772 8536 ts_tablet_manager.cc:1081] T 3a7d8f424df742599d3f6d48d9b1a37d P 3ce79ee2bb1348d6a36d8ce39df7d3aa: Time spent starting tablet: real 0.009s user 0.008s sys 0.000s I0507 06:32:03.571815 8535 raft_consensus.cc:340] T 3a7d8f424df742599d3f6d48d9b1a37d P 2781869a927549908dde07bc5019d0e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ce79ee2bb1348d6a36d8ce39df7d3aa" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 44481 } } peers { permanent_uuid: "d9fd45507a8e435abc710b00a27538c6" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 44183 } } peers { permanent_uuid: "2781869a927549908dde07bc5019d0e4" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 40501 } } I0507 06:32:03.572146 8535 raft_consensus.cc:366] T 3a7d8f424df742599d3f6d48d9b1a37d P 2781869a927549908dde07bc5019d0e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0507 06:32:03.572199 8535 raft_consensus.cc:705] T 3a7d8f424df742599d3f6d48d9b1a37d P 2781869a927549908dde07bc5019d0e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2781869a927549908dde07bc5019d0e4, State: Initialized, Role: FOLLOWER I0507 06:32:03.572649 8535 consensus_queue.cc:229] T 3a7d8f424df742599d3f6d48d9b1a37d P 2781869a927549908dde07bc5019d0e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ce79ee2bb1348d6a36d8ce39df7d3aa" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 44481 } } peers { permanent_uuid: "d9fd45507a8e435abc710b00a27538c6" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 44183 } } peers { permanent_uuid: "2781869a927549908dde07bc5019d0e4" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 40501 } } I0507 06:32:03.573709 8535 ts_tablet_manager.cc:1081] T 3a7d8f424df742599d3f6d48d9b1a37d P 2781869a927549908dde07bc5019d0e4: Time spent starting tablet: real 0.008s user 0.003s sys 0.003s W0507 06:32:03.606544 8027 tablet.cc:1786] T 3a7d8f424df742599d3f6d48d9b1a37d P 2781869a927549908dde07bc5019d0e4: Can't schedule compaction. Clean time has not been advanced past its initial value. I0507 06:32:03.633074 8148 heartbeater.cc:467] Master 127.0.57.254:34231 was elected leader, sending a full tablet report... W0507 06:32:03.779695 7785 tablet.cc:1786] T 3a7d8f424df742599d3f6d48d9b1a37d P 3ce79ee2bb1348d6a36d8ce39df7d3aa: Can't schedule compaction. Clean time has not been advanced past its initial value. W0507 06:32:03.801656 7905 tablet.cc:1786] T 3a7d8f424df742599d3f6d48d9b1a37d P d9fd45507a8e435abc710b00a27538c6: Can't schedule compaction. Clean time has not been advanced past its initial value. I0507 06:32:03.846190 8540 raft_consensus.cc:472] T 3a7d8f424df742599d3f6d48d9b1a37d P d9fd45507a8e435abc710b00a27538c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0507 06:32:03.846480 8540 raft_consensus.cc:494] T 3a7d8f424df742599d3f6d48d9b1a37d P d9fd45507a8e435abc710b00a27538c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ce79ee2bb1348d6a36d8ce39df7d3aa" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 44481 } } peers { permanent_uuid: "d9fd45507a8e435abc710b00a27538c6" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 44183 } } peers { permanent_uuid: "2781869a927549908dde07bc5019d0e4" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 40501 } } I0507 06:32:03.848101 8540 leader_election.cc:296] T 3a7d8f424df742599d3f6d48d9b1a37d P d9fd45507a8e435abc710b00a27538c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3ce79ee2bb1348d6a36d8ce39df7d3aa (127.0.57.193:44481), 2781869a927549908dde07bc5019d0e4 (127.0.57.195:40501) I0507 06:32:03.854157 7738 tablet_service.cc:1286] Received RequestConsensusVote() RPC: tablet_id: "3a7d8f424df742599d3f6d48d9b1a37d" candidate_uuid: "d9fd45507a8e435abc710b00a27538c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ce79ee2bb1348d6a36d8ce39df7d3aa" is_pre_election: true I0507 06:32:03.854548 7738 raft_consensus.cc:2346] T 3a7d8f424df742599d3f6d48d9b1a37d P 3ce79ee2bb1348d6a36d8ce39df7d3aa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9fd45507a8e435abc710b00a27538c6 in term 0. I0507 06:32:03.854465 7983 tablet_service.cc:1286] Received RequestConsensusVote() RPC: tablet_id: "3a7d8f424df742599d3f6d48d9b1a37d" candidate_uuid: "d9fd45507a8e435abc710b00a27538c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2781869a927549908dde07bc5019d0e4" is_pre_election: true I0507 06:32:03.854852 7983 raft_consensus.cc:2346] T 3a7d8f424df742599d3f6d48d9b1a37d P 2781869a927549908dde07bc5019d0e4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9fd45507a8e435abc710b00a27538c6 in term 0. I0507 06:32:03.855213 7792 leader_election.cc:310] T 3a7d8f424df742599d3f6d48d9b1a37d P d9fd45507a8e435abc710b00a27538c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ce79ee2bb1348d6a36d8ce39df7d3aa, d9fd45507a8e435abc710b00a27538c6; no voters: I0507 06:32:03.855729 8540 raft_consensus.cc:2640] T 3a7d8f424df742599d3f6d48d9b1a37d P d9fd45507a8e435abc710b00a27538c6 [term 0 FOLLOWER]: Leader pre-election won for term 1 I0507 06:32:03.855856 8540 raft_consensus.cc:472] T 3a7d8f424df742599d3f6d48d9b1a37d P d9fd45507a8e435abc710b00a27538c6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0507 06:32:03.855909 8540 raft_consensus.cc:2886] T 3a7d8f424df742599d3f6d48d9b1a37d P d9fd45507a8e435abc710b00a27538c6 [term 0 FOLLOWER]: Advancing to term 1 I0507 06:32:03.857753 8540 raft_consensus.cc:494] T 3a7d8f424df742599d3f6d48d9b1a37d P d9fd45507a8e435abc710b00a27538c6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ce79ee2bb1348d6a36d8ce39df7d3aa" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 44481 } } peers { permanent_uuid: "d9fd45507a8e435abc710b00a27538c6" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 44183 } } peers { permanent_uuid: "2781869a927549908dde07bc5019d0e4" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 40501 } } I0507 06:32:03.858353 8540 leader_election.cc:296] T 3a7d8f424df742599d3f6d48d9b1a37d P d9fd45507a8e435abc710b00a27538c6 [CANDIDATE]: Term 1 election: Requested vote from peers 3ce79ee2bb1348d6a36d8ce39df7d3aa (127.0.57.193:44481), 2781869a927549908dde07bc5019d0e4 (127.0.57.195:40501) I0507 06:32:03.858929 7983 tablet_service.cc:1286] Received RequestConsensusVote() RPC: tablet_id: "3a7d8f424df742599d3f6d48d9b1a37d" candidate_uuid: "d9fd45507a8e435abc710b00a27538c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2781869a927549908dde07bc5019d0e4" I0507 06:32:03.858899 7738 tablet_service.cc:1286] Received RequestConsensusVote() RPC: tablet_id: "3a7d8f424df742599d3f6d48d9b1a37d" candidate_uuid: "d9fd45507a8e435abc710b00a27538c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ce79ee2bb1348d6a36d8ce39df7d3aa" I0507 06:32:03.859068 7983 raft_consensus.cc:2886] T 3a7d8f424df742599d3f6d48d9b1a37d P 2781869a927549908dde07bc5019d0e4 [term 0 FOLLOWER]: Advancing to term 1 I0507 06:32:03.859072 7738 raft_consensus.cc:2886] T 3a7d8f424df742599d3f6d48d9b1a37d P 3ce79ee2bb1348d6a36d8ce39df7d3aa [term 0 FOLLOWER]: Advancing to term 1 I0507 06:32:03.860970 7983 raft_consensus.cc:2346] T 3a7d8f424df742599d3f6d48d9b1a37d P 2781869a927549908dde07bc5019d0e4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9fd45507a8e435abc710b00a27538c6 in term 1. I0507 06:32:03.861030 7738 raft_consensus.cc:2346] T 3a7d8f424df742599d3f6d48d9b1a37d P 3ce79ee2bb1348d6a36d8ce39df7d3aa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9fd45507a8e435abc710b00a27538c6 in term 1. I0507 06:32:03.861361 7794 leader_election.cc:310] T 3a7d8f424df742599d3f6d48d9b1a37d P d9fd45507a8e435abc710b00a27538c6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2781869a927549908dde07bc5019d0e4, d9fd45507a8e435abc710b00a27538c6; no voters: I0507 06:32:03.861726 8540 raft_consensus.cc:2640] T 3a7d8f424df742599d3f6d48d9b1a37d P d9fd45507a8e435abc710b00a27538c6 [term 1 FOLLOWER]: Leader election won for term 1 I0507 06:32:03.862264 8540 raft_consensus.cc:667] T 3a7d8f424df742599d3f6d48d9b1a37d P d9fd45507a8e435abc710b00a27538c6 [term 1 LEADER]: Becoming Leader. State: Replica: d9fd45507a8e435abc710b00a27538c6, State: Running, Role: LEADER I0507 06:32:03.862674 8540 consensus_queue.cc:206] T 3a7d8f424df742599d3f6d48d9b1a37d P d9fd45507a8e435abc710b00a27538c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ce79ee2bb1348d6a36d8ce39df7d3aa" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 44481 } } peers { permanent_uuid: "d9fd45507a8e435abc710b00a27538c6" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 44183 } } peers { permanent_uuid: "2781869a927549908dde07bc5019d0e4" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 40501 } } I0507 06:32:03.867611 7617 catalog_manager.cc:3988] T 3a7d8f424df742599d3f6d48d9b1a37d P d9fd45507a8e435abc710b00a27538c6 reported cstate change: term changed from 0 to 1, leader changed from to d9fd45507a8e435abc710b00a27538c6 (127.0.57.194). New cstate: current_term: 1 leader_uuid: "d9fd45507a8e435abc710b00a27538c6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ce79ee2bb1348d6a36d8ce39df7d3aa" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 44481 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d9fd45507a8e435abc710b00a27538c6" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 44183 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2781869a927549908dde07bc5019d0e4" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 40501 } health_report { overall_health: UNKNOWN } } } I0507 06:32:03.898910 231 external_mini_cluster.cc:623] 7 TS(s) registered with all masters I0507 06:32:03.900596 231 ts_itest-base.cc:232] Waiting for 1 tablets on tserver 3ce79ee2bb1348d6a36d8ce39df7d3aa to finish bootstrapping I0507 06:32:03.907709 231 ts_itest-base.cc:232] Waiting for 1 tablets on tserver d9fd45507a8e435abc710b00a27538c6 to finish bootstrapping I0507 06:32:03.914412 231 ts_itest-base.cc:232] Waiting for 1 tablets on tserver 2781869a927549908dde07bc5019d0e4 to finish bootstrapping I0507 06:32:03.918479 8270 heartbeater.cc:467] Master 127.0.57.254:34231 was elected leader, sending a full tablet report... I0507 06:32:03.920887 231 ts_itest-base.cc:232] Waiting for 0 tablets on tserver b5c46129f1994ab28f3710e8e392ceab to finish bootstrapping I0507 06:32:03.928295 231 ts_itest-base.cc:232] Waiting for 0 tablets on tserver bb5a49b22d4048608d9e7b762d749fb2 to finish bootstrapping I0507 06:32:03.935091 231 ts_itest-base.cc:232] Waiting for 0 tablets on tserver 3e033fe6ec6f4fe595bdcd8a95a9f4e8 to finish bootstrapping I0507 06:32:03.941890 231 ts_itest-base.cc:232] Waiting for 0 tablets on tserver 44a1757d4e604357a1991e808f009e56 to finish bootstrapping I0507 06:32:03.949080 231 test_util.cc:212] Using random seed: 1611307523 I0507 06:32:03.960834 231 test_workload.cc:306] TestWorkload: Skipping table creation because table TestTable already exists I0507 06:32:04.001185 7983 raft_consensus.cc:1181] T 3a7d8f424df742599d3f6d48d9b1a37d P 2781869a927549908dde07bc5019d0e4 [term 1 FOLLOWER]: Refusing update from remote peer d9fd45507a8e435abc710b00a27538c6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0507 06:32:04.001408 7738 raft_consensus.cc:1181] T 3a7d8f424df742599d3f6d48d9b1a37d P 3ce79ee2bb1348d6a36d8ce39df7d3aa [term 1 FOLLOWER]: Refusing update from remote peer d9fd45507a8e435abc710b00a27538c6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0507 06:32:04.002513 8540 consensus_queue.cc:984] T 3a7d8f424df742599d3f6d48d9b1a37d P d9fd45507a8e435abc710b00a27538c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2781869a927549908dde07bc5019d0e4" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 40501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0507 06:32:04.002833 8545 consensus_queue.cc:984] T 3a7d8f424df742599d3f6d48d9b1a37d P d9fd45507a8e435abc710b00a27538c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ce79ee2bb1348d6a36d8ce39df7d3aa" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 44481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0507 06:32:04.018188 8561 mvcc.cc:203] Tried to move safe_time back from 6378335125498007552 to 6378335124944687104. Current Snapshot: MvccSnapshot[committed={T|T < 6378335125498007552}] I0507 06:32:04.042414 8563 mvcc.cc:203] Tried to move safe_time back from 6378335125507919872 to 6378335124944687104. Current Snapshot: MvccSnapshot[committed={T|T < 6378335125507919872}] I0507 06:32:04.044433 8562 mvcc.cc:203] Tried to move safe_time back from 6378335125498007552 to 6378335124944687104. Current Snapshot: MvccSnapshot[committed={T|T < 6378335125498007552}] I0507 06:32:04.061733 231 external_mini_cluster.cc:1172] Attempting to check leaks for /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver pid 7907 I0507 06:32:04.062686 8023 generic_service.cc:171] Checking for leaks (request via RPC) I0507 06:32:04.113945 231 external_mini_cluster.cc:1131] Killing /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver with pid 7907 I0507 06:32:04.207170 8392 heartbeater.cc:467] Master 127.0.57.254:34231 was elected leader, sending a full tablet report... W0507 06:32:04.473095 7794 consensus_peers.cc:457] T 3a7d8f424df742599d3f6d48d9b1a37d P d9fd45507a8e435abc710b00a27538c6 -> Peer 2781869a927549908dde07bc5019d0e4 (127.0.57.195:40501): Couldn't send request to peer 2781869a927549908dde07bc5019d0e4. Status: Network error: Client connection negotiation failed: client connection to 127.0.57.195:40501: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry. I0507 06:32:04.490702 8514 heartbeater.cc:467] Master 127.0.57.254:34231 was elected leader, sending a full tablet report... I0507 06:32:06.520694 8568 consensus_queue.cc:544] T 3a7d8f424df742599d3f6d48d9b1a37d P d9fd45507a8e435abc710b00a27538c6 [LEADER]: Leader has been unable to successfully communicate with peer 2781869a927549908dde07bc5019d0e4 for more than 2 seconds (2.459s) I0507 06:32:06.523277 7608 catalog_manager.cc:3616] Sending ChangeConfig:ADD_PEER:NON_VOTER on tablet 3a7d8f424df742599d3f6d48d9b1a37d (attempt 1) I0507 06:32:06.525537 7861 consensus_queue.cc:206] T 3a7d8f424df742599d3f6d48d9b1a37d P d9fd45507a8e435abc710b00a27538c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3ce79ee2bb1348d6a36d8ce39df7d3aa" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 44481 } } peers { permanent_uuid: "d9fd45507a8e435abc710b00a27538c6" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 44183 } } peers { permanent_uuid: "2781869a927549908dde07bc5019d0e4" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 40501 } } peers { permanent_uuid: "b5c46129f1994ab28f3710e8e392ceab" member_type: NON_VOTER last_known_addr { host: "127.0.57.196" port: 34535 } attrs { promote: true } } I0507 06:32:06.527905 7738 raft_consensus.cc:1181] T 3a7d8f424df742599d3f6d48d9b1a37d P 3ce79ee2bb1348d6a36d8ce39df7d3aa [term 1 FOLLOWER]: Refusing update from remote peer d9fd45507a8e435abc710b00a27538c6: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch) W0507 06:32:06.528034 7794 consensus_peers.cc:457] T 3a7d8f424df742599d3f6d48d9b1a37d P d9fd45507a8e435abc710b00a27538c6 -> Peer 2781869a927549908dde07bc5019d0e4 (127.0.57.195:40501): Couldn't send request to peer 2781869a927549908dde07bc5019d0e4. Status: Network error: Client connection negotiation failed: client connection to 127.0.57.195:40501: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry. I0507 06:32:06.528445 8582 consensus_queue.cc:984] T 3a7d8f424df742599d3f6d48d9b1a37d P d9fd45507a8e435abc710b00a27538c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ce79ee2bb1348d6a36d8ce39df7d3aa" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 44481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s I0507 06:32:06.531062 8582 raft_consensus.cc:2788] T 3a7d8f424df742599d3f6d48d9b1a37d P d9fd45507a8e435abc710b00a27538c6 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index -1 to 8, NON_VOTER b5c46129f1994ab28f3710e8e392ceab (127.0.57.196) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3ce79ee2bb1348d6a36d8ce39df7d3aa" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 44481 } } peers { permanent_uuid: "d9fd45507a8e435abc710b00a27538c6" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 44183 } } peers { permanent_uuid: "2781869a927549908dde07bc5019d0e4" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 40501 } } peers { permanent_uuid: "b5c46129f1994ab28f3710e8e392ceab" member_type: NON_VOTER last_known_addr { host: "127.0.57.196" port: 34535 } attrs { promote: true } } } I0507 06:32:06.531884 7738 raft_consensus.cc:2788] T 3a7d8f424df742599d3f6d48d9b1a37d P 3ce79ee2bb1348d6a36d8ce39df7d3aa [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index -1 to 8, NON_VOTER b5c46129f1994ab28f3710e8e392ceab (127.0.57.196) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3ce79ee2bb1348d6a36d8ce39df7d3aa" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 44481 } } peers { permanent_uuid: "d9fd45507a8e435abc710b00a27538c6" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 44183 } } peers { permanent_uuid: "2781869a927549908dde07bc5019d0e4" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 40501 } } peers { permanent_uuid: "b5c46129f1994ab28f3710e8e392ceab" member_type: NON_VOTER last_known_addr { host: "127.0.57.196" port: 34535 } attrs { promote: true } } } I0507 06:32:06.533902 7599 catalog_manager.cc:3531] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 3a7d8f424df742599d3f6d48d9b1a37d with cas_config_opid_index -1: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1) W0507 06:32:06.534749 7795 consensus_peers.cc:457] T 3a7d8f424df742599d3f6d48d9b1a37d P d9fd45507a8e435abc710b00a27538c6 -> Peer b5c46129f1994ab28f3710e8e392ceab (127.0.57.196:34535): Couldn't send request to peer b5c46129f1994ab28f3710e8e392ceab. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3a7d8f424df742599d3f6d48d9b1a37d. This is attempt 1: this message will repeat every 5th retry. I0507 06:32:06.535068 7608 catalog_manager.cc:3988] T 3a7d8f424df742599d3f6d48d9b1a37d P d9fd45507a8e435abc710b00a27538c6 reported cstate change: config changed from index -1 to 8, NON_VOTER b5c46129f1994ab28f3710e8e392ceab (127.0.57.196) added. New cstate: current_term: 1 leader_uuid: "d9fd45507a8e435abc710b00a27538c6" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3ce79ee2bb1348d6a36d8ce39df7d3aa" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 44481 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d9fd45507a8e435abc710b00a27538c6" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 44183 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2781869a927549908dde07bc5019d0e4" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 40501 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b5c46129f1994ab28f3710e8e392ceab" member_type: NON_VOTER last_known_addr { host: "127.0.57.196" port: 34535 } attrs { promote: true } health_report { overall_health: UNKNOWN } } } I0507 06:32:06.558207 231 external_mini_cluster.cc:1172] Attempting to check leaks for /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver pid 7651 I0507 06:32:06.559015 7778 generic_service.cc:171] Checking for leaks (request via RPC) I0507 06:32:06.610193 231 external_mini_cluster.cc:1131] Killing /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver with pid 7651 I0507 06:32:06.944396 8596 ts_tablet_manager.cc:676] T 3a7d8f424df742599d3f6d48d9b1a37d P b5c46129f1994ab28f3710e8e392ceab: Initiating tablet copy from peer d9fd45507a8e435abc710b00a27538c6 (127.0.57.194:44183) I0507 06:32:06.944934 8596 tablet_copy_client.cc:240] T 3a7d8f424df742599d3f6d48d9b1a37d P b5c46129f1994ab28f3710e8e392ceab: tablet copy: Beginning tablet copy session from remote peer at address 127.0.57.194:44183 I0507 06:32:06.951246 7881 tablet_copy_service.cc:135] P d9fd45507a8e435abc710b00a27538c6: Received BeginTabletCopySession request for tablet 3a7d8f424df742599d3f6d48d9b1a37d from peer b5c46129f1994ab28f3710e8e392ceab ({username='slave'} at 127.0.57.196:42593) I0507 06:32:06.951453 7881 tablet_copy_service.cc:156] P d9fd45507a8e435abc710b00a27538c6: Beginning new tablet copy session on tablet 3a7d8f424df742599d3f6d48d9b1a37d from peer b5c46129f1994ab28f3710e8e392ceab at {username='slave'} at 127.0.57.196:42593: session id = b5c46129f1994ab28f3710e8e392ceab-3a7d8f424df742599d3f6d48d9b1a37d I0507 06:32:06.953732 7881 tablet_copy_source_session.cc:202] T 3a7d8f424df742599d3f6d48d9b1a37d P d9fd45507a8e435abc710b00a27538c6: Tablet Copy: opened 0 blocks and 1 log segments I0507 06:32:06.957336 8596 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet 3a7d8f424df742599d3f6d48d9b1a37d. 1 dirs total, 0 dirs full, 0 dirs failed I0507 06:32:06.967447 8596 tablet_copy_client.cc:575] T 3a7d8f424df742599d3f6d48d9b1a37d P b5c46129f1994ab28f3710e8e392ceab: tablet copy: Starting download of 0 data blocks... I0507 06:32:06.967630 8596 tablet_copy_client.cc:538] T 3a7d8f424df742599d3f6d48d9b1a37d P b5c46129f1994ab28f3710e8e392ceab: tablet copy: Starting download of 1 WAL segments... I0507 06:32:06.969488 8596 tablet_copy_client.cc:749] T 3a7d8f424df742599d3f6d48d9b1a37d P b5c46129f1994ab28f3710e8e392ceab: tablet copy: Injecting latency into file download: 10000 W0507 06:32:07.101197 7792 consensus_peers.cc:457] T 3a7d8f424df742599d3f6d48d9b1a37d P d9fd45507a8e435abc710b00a27538c6 -> Peer 3ce79ee2bb1348d6a36d8ce39df7d3aa (127.0.57.193:44481): Couldn't send request to peer 3ce79ee2bb1348d6a36d8ce39df7d3aa. Status: Network error: Client connection negotiation failed: client connection to 127.0.57.193:44481: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry. I0507 06:32:08.587546 8601 consensus_queue.cc:544] T 3a7d8f424df742599d3f6d48d9b1a37d P d9fd45507a8e435abc710b00a27538c6 [LEADER]: Leader has been unable to successfully communicate with peer 3ce79ee2bb1348d6a36d8ce39df7d3aa for more than 2 seconds (2.052s) I0507 06:32:08.944341 8601 consensus_queue.cc:544] T 3a7d8f424df742599d3f6d48d9b1a37d P d9fd45507a8e435abc710b00a27538c6 [LEADER]: Leader has been unable to successfully communicate with peer 2781869a927549908dde07bc5019d0e4 for more than 2 seconds (2.418s) W0507 06:32:08.945510 7794 consensus_peers.cc:457] T 3a7d8f424df742599d3f6d48d9b1a37d P d9fd45507a8e435abc710b00a27538c6 -> Peer 2781869a927549908dde07bc5019d0e4 (127.0.57.195:40501): Couldn't send request to peer 2781869a927549908dde07bc5019d0e4. Status: Network error: Client connection negotiation failed: client connection to 127.0.57.195:40501: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry. W0507 06:32:09.618470 7792 consensus_peers.cc:457] T 3a7d8f424df742599d3f6d48d9b1a37d P d9fd45507a8e435abc710b00a27538c6 -> Peer 3ce79ee2bb1348d6a36d8ce39df7d3aa (127.0.57.193:44481): Couldn't send request to peer 3ce79ee2bb1348d6a36d8ce39df7d3aa. Status: Network error: Client connection negotiation failed: client connection to 127.0.57.193:44481: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry. W0507 06:32:09.638504 7795 consensus_peers.cc:457] T 3a7d8f424df742599d3f6d48d9b1a37d P d9fd45507a8e435abc710b00a27538c6 -> Peer b5c46129f1994ab28f3710e8e392ceab (127.0.57.196:34535): Couldn't send request to peer b5c46129f1994ab28f3710e8e392ceab. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 6: this message will repeat every 5th retry. W0507 06:32:11.617746 7794 consensus_peers.cc:457] T 3a7d8f424df742599d3f6d48d9b1a37d P d9fd45507a8e435abc710b00a27538c6 -> Peer 2781869a927549908dde07bc5019d0e4 (127.0.57.195:40501): Couldn't send request to peer 2781869a927549908dde07bc5019d0e4. Status: Network error: Client connection negotiation failed: client connection to 127.0.57.195:40501: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry. W0507 06:32:12.139991 7792 consensus_peers.cc:457] T 3a7d8f424df742599d3f6d48d9b1a37d P d9fd45507a8e435abc710b00a27538c6 -> Peer 3ce79ee2bb1348d6a36d8ce39df7d3aa (127.0.57.193:44481): Couldn't send request to peer 3ce79ee2bb1348d6a36d8ce39df7d3aa. Status: Network error: Client connection negotiation failed: client connection to 127.0.57.193:44481: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry. W0507 06:32:12.504300 7795 consensus_peers.cc:457] T 3a7d8f424df742599d3f6d48d9b1a37d P d9fd45507a8e435abc710b00a27538c6 -> Peer b5c46129f1994ab28f3710e8e392ceab (127.0.57.196:34535): Couldn't send request to peer b5c46129f1994ab28f3710e8e392ceab. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 11: this message will repeat every 5th retry. W0507 06:32:14.169613 7794 consensus_peers.cc:457] T 3a7d8f424df742599d3f6d48d9b1a37d P d9fd45507a8e435abc710b00a27538c6 -> Peer 2781869a927549908dde07bc5019d0e4 (127.0.57.195:40501): Couldn't send request to peer 2781869a927549908dde07bc5019d0e4. Status: Network error: Client connection negotiation failed: client connection to 127.0.57.195:40501: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry. W0507 06:32:14.720439 7795 consensus_peers.cc:457] T 3a7d8f424df742599d3f6d48d9b1a37d P d9fd45507a8e435abc710b00a27538c6 -> Peer b5c46129f1994ab28f3710e8e392ceab (127.0.57.196:34535): Couldn't send request to peer b5c46129f1994ab28f3710e8e392ceab. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 16: this message will repeat every 5th retry. W0507 06:32:14.736371 7792 consensus_peers.cc:457] T 3a7d8f424df742599d3f6d48d9b1a37d P d9fd45507a8e435abc710b00a27538c6 -> Peer 3ce79ee2bb1348d6a36d8ce39df7d3aa (127.0.57.193:44481): Couldn't send request to peer 3ce79ee2bb1348d6a36d8ce39df7d3aa. Status: Network error: Client connection negotiation failed: client connection to 127.0.57.193:44481: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry. I0507 06:32:16.969954 8596 tablet_copy_client.cc:414] T 3a7d8f424df742599d3f6d48d9b1a37d P b5c46129f1994ab28f3710e8e392ceab: tablet copy: Tablet Copy complete. Replacing tablet superblock. I0507 06:32:16.973134 8596 tablet_bootstrap.cc:439] T 3a7d8f424df742599d3f6d48d9b1a37d P b5c46129f1994ab28f3710e8e392ceab: Bootstrap starting. I0507 06:32:16.980059 8596 log.cc:525] T 3a7d8f424df742599d3f6d48d9b1a37d P b5c46129f1994ab28f3710e8e392ceab: Log is configured to *not* fsync() on all Append() calls I0507 06:32:17.002893 8596 tablet_bootstrap.cc:439] T 3a7d8f424df742599d3f6d48d9b1a37d P b5c46129f1994ab28f3710e8e392ceab: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=300 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I0507 06:32:17.003345 8596 tablet_bootstrap.cc:439] T 3a7d8f424df742599d3f6d48d9b1a37d P b5c46129f1994ab28f3710e8e392ceab: Bootstrap complete. I0507 06:32:17.003688 8596 ts_tablet_manager.cc:1053] T 3a7d8f424df742599d3f6d48d9b1a37d P b5c46129f1994ab28f3710e8e392ceab: Time spent bootstrapping tablet: real 0.031s user 0.023s sys 0.004s I0507 06:32:17.008224 8596 raft_consensus.cc:340] T 3a7d8f424df742599d3f6d48d9b1a37d P b5c46129f1994ab28f3710e8e392ceab [term 1 LEARNER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3ce79ee2bb1348d6a36d8ce39df7d3aa" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 44481 } } peers { permanent_uuid: "d9fd45507a8e435abc710b00a27538c6" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 44183 } } peers { permanent_uuid: "2781869a927549908dde07bc5019d0e4" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 40501 } } peers { permanent_uuid: "b5c46129f1994ab28f3710e8e392ceab" member_type: NON_VOTER last_known_addr { host: "127.0.57.196" port: 34535 } attrs { promote: true } } I0507 06:32:17.008666 8596 raft_consensus.cc:705] T 3a7d8f424df742599d3f6d48d9b1a37d P b5c46129f1994ab28f3710e8e392ceab [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b5c46129f1994ab28f3710e8e392ceab, State: Initialized, Role: LEARNER I0507 06:32:17.009001 8596 consensus_queue.cc:229] T 3a7d8f424df742599d3f6d48d9b1a37d P b5c46129f1994ab28f3710e8e392ceab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3ce79ee2bb1348d6a36d8ce39df7d3aa" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 44481 } } peers { permanent_uuid: "d9fd45507a8e435abc710b00a27538c6" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 44183 } } peers { permanent_uuid: "2781869a927549908dde07bc5019d0e4" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 40501 } } peers { permanent_uuid: "b5c46129f1994ab28f3710e8e392ceab" member_type: NON_VOTER last_known_addr { host: "127.0.57.196" port: 34535 } attrs { promote: true } } I0507 06:32:17.011229 8596 ts_tablet_manager.cc:1081] T 3a7d8f424df742599d3f6d48d9b1a37d P b5c46129f1994ab28f3710e8e392ceab: Time spent starting tablet: real 0.007s user 0.009s sys 0.000s I0507 06:32:17.011605 231 external_mini_cluster.cc:1172] Attempting to check leaks for /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver pid 7782 I0507 06:32:17.012094 7881 tablet_copy_service.cc:325] P d9fd45507a8e435abc710b00a27538c6: Request end of tablet copy session b5c46129f1994ab28f3710e8e392ceab-3a7d8f424df742599d3f6d48d9b1a37d received from {username='slave'} at 127.0.57.196:42593 I0507 06:32:17.012228 7881 tablet_copy_service.cc:413] P d9fd45507a8e435abc710b00a27538c6: ending tablet copy session b5c46129f1994ab28f3710e8e392ceab-3a7d8f424df742599d3f6d48d9b1a37d on tablet 3a7d8f424df742599d3f6d48d9b1a37d with peer b5c46129f1994ab28f3710e8e392ceab I0507 06:32:17.012603 7901 generic_service.cc:171] Checking for leaks (request via RPC) W0507 06:32:17.067843 7794 consensus_peers.cc:457] T 3a7d8f424df742599d3f6d48d9b1a37d P d9fd45507a8e435abc710b00a27538c6 -> Peer 2781869a927549908dde07bc5019d0e4 (127.0.57.195:40501): Couldn't send request to peer 2781869a927549908dde07bc5019d0e4. Status: Network error: Client connection negotiation failed: client connection to 127.0.57.195:40501: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry. I0507 06:32:17.067976 231 external_mini_cluster.cc:1131] Killing /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver with pid 7782 I0507 06:32:17.080513 231 external_mini_cluster.cc:1172] Attempting to check leaks for /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver pid 8029 I0507 06:32:17.081720 8145 generic_service.cc:171] Checking for leaks (request via RPC) I0507 06:32:17.150036 231 external_mini_cluster.cc:1131] Killing /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver with pid 8029 I0507 06:32:17.161512 231 external_mini_cluster.cc:1172] Attempting to check leaks for /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver pid 8151 I0507 06:32:17.162722 8267 generic_service.cc:171] Checking for leaks (request via RPC) I0507 06:32:17.208273 231 external_mini_cluster.cc:1131] Killing /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver with pid 8151 I0507 06:32:17.218312 231 external_mini_cluster.cc:1172] Attempting to check leaks for /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver pid 8273 I0507 06:32:17.219528 8389 generic_service.cc:171] Checking for leaks (request via RPC) I0507 06:32:17.269508 231 external_mini_cluster.cc:1131] Killing /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver with pid 8273 W0507 06:32:17.278836 7599 connection.cc:647] server connection from 127.0.57.198:40531 recv error: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107) I0507 06:32:17.279062 231 external_mini_cluster.cc:1172] Attempting to check leaks for /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver pid 8395 I0507 06:32:17.280302 8511 generic_service.cc:171] Checking for leaks (request via RPC) I0507 06:32:17.325572 231 external_mini_cluster.cc:1131] Killing /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver with pid 8395 I0507 06:32:17.335122 231 external_mini_cluster.cc:1172] Attempting to check leaks for /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-master pid 7591 I0507 06:32:17.336282 7647 generic_service.cc:171] Checking for leaks (request via RPC) I0507 06:32:17.415135 231 external_mini_cluster.cc:1131] Killing /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-master with pid 7591 I0507 06:32:17.430747 231 external_mini_cluster.cc:933] Running /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.RestartClusterWithNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-0/wal --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.RestartClusterWithNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data --block_manager=log --rpc_bind_addresses=127.0.57.193:44481 --local_ip_for_outbound_sockets=127.0.57.193 --webserver_port=44839 --webserver_interface=127.0.57.193 --tserver_master_addrs=127.0.57.254:34231 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.RestartClusterWithNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-0/logs --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.RestartClusterWithNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=true --tablet_copy_download_file_inject_latency_ms=10000 --follower_unavailable_considered_failed_sec=2 --consensus_rpc_timeout_ms=5000 with env {} WARNING: Logging before InitGoogleLogging() is written to STDERR W0507 06:32:17.609709 8615 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0507 06:32:17.609989 8615 flags.cc:404] Enabled unsafe flag: --never_fsync=true W0507 06:32:17.612157 8615 flags.cc:404] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true W0507 06:32:17.612222 8615 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W0507 06:32:17.612354 8615 flags.cc:404] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.57.193 I0507 06:32:17.616793 8615 tablet_server_main.cc:78] Tablet server non-default flags: --consensus_rpc_timeout_ms=5000 --follower_unavailable_considered_failed_sec=2 --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.RestartClusterWithNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.RestartClusterWithNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-0/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.0.57.193:44481 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.RestartClusterWithNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data/info.pb --webserver_interface=127.0.57.193 --webserver_port=44839 --tablet_copy_download_file_inject_latency_ms=10000 --tserver_master_addrs=127.0.57.254:34231 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.0.57.193 --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.RestartClusterWithNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-0/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.10.0-cdh6.x-SNAPSHOT revision 7d0ee26697926e87af9ef395008799577f6ab43f build type FASTDEBUG built by jenkins at 06 May 2019 22:44:21 PST on quasar-knmpfx-1.vpc.cloudera.com build id 322017 ASAN enabled I0507 06:32:17.625105 8615 system_ntp.cc:202] NTP initialized. Skew: 500ppm Current error: 143016us I0507 06:32:17.625918 8615 fs_manager.cc:263] Metadata directory not provided I0507 06:32:17.626006 8615 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0507 06:32:17.632459 8615 fs_manager.cc:399] Time spent opening directory manager: real 0.005s user 0.006s sys 0.000s I0507 06:32:17.633203 8615 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0507 06:32:17.633852 8615 file_cache.cc:466] Constructed file cache lbm with capacity 419430 I0507 06:32:17.637390 8615 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s I0507 06:32:17.637493 8615 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.RestartClusterWithNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.RestartClusterWithNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-0/wal uuid: "3ce79ee2bb1348d6a36d8ce39df7d3aa" format_stamp: "Formatted at 2019-05-07 06:32:01 on dist-test-slave-dist-test-slave-rqp6" I0507 06:32:17.638535 8615 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.RestartClusterWithNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-0/wal metadata directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.RestartClusterWithNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-0/wal 1 data directories: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.RestartClusterWithNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0507 06:32:17.657716 8615 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0507 06:32:17.659718 8615 ts_tablet_manager.cc:344] Loading tablet metadata (0/1 complete) I0507 06:32:17.663962 8615 ts_tablet_manager.cc:356] Loaded tablet metadata (1 total tablets, 1 live tablets) I0507 06:32:17.664021 8615 ts_tablet_manager.cc:362] Registering tablets (0/1 complete) I0507 06:32:17.666972 8615 ts_tablet_manager.cc:376] Registered 1 tablets I0507 06:32:17.667714 8630 tablet_bootstrap.cc:439] T 3a7d8f424df742599d3f6d48d9b1a37d P 3ce79ee2bb1348d6a36d8ce39df7d3aa: Bootstrap starting. I0507 06:32:17.684809 8630 log.cc:525] T 3a7d8f424df742599d3f6d48d9b1a37d P 3ce79ee2bb1348d6a36d8ce39df7d3aa: Log is configured to *not* fsync() on all Append() calls I0507 06:32:17.718150 8630 tablet_bootstrap.cc:439] T 3a7d8f424df742599d3f6d48d9b1a37d P 3ce79ee2bb1348d6a36d8ce39df7d3aa: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=300 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I0507 06:32:17.718874 8630 tablet_bootstrap.cc:439] T 3a7d8f424df742599d3f6d48d9b1a37d P 3ce79ee2bb1348d6a36d8ce39df7d3aa: Bootstrap complete. I0507 06:32:17.720513 8630 ts_tablet_manager.cc:1053] T 3a7d8f424df742599d3f6d48d9b1a37d P 3ce79ee2bb1348d6a36d8ce39df7d3aa: Time spent bootstrapping tablet: real 0.053s user 0.040s sys 0.008s I0507 06:32:17.733270 8615 rpc_server.cc:205] RPC server started. Bound to: 127.0.57.193:44481 I0507 06:32:17.735064 8615 webserver.cc:308] Webserver started at http://127.0.57.193:44839/ using document root and password file I0507 06:32:17.731935 8630 raft_consensus.cc:340] T 3a7d8f424df742599d3f6d48d9b1a37d P 3ce79ee2bb1348d6a36d8ce39df7d3aa [term 1 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3ce79ee2bb1348d6a36d8ce39df7d3aa" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 44481 } } peers { permanent_uuid: "d9fd45507a8e435abc710b00a27538c6" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 44183 } } peers { permanent_uuid: "2781869a927549908dde07bc5019d0e4" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 40501 } } peers { permanent_uuid: "b5c46129f1994ab28f3710e8e392ceab" member_type: NON_VOTER last_known_addr { host: "127.0.57.196" port: 34535 } attrs { promote: true } } I0507 06:32:17.735713 8630 raft_consensus.cc:705] T 3a7d8f424df742599d3f6d48d9b1a37d P 3ce79ee2bb1348d6a36d8ce39df7d3aa [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ce79ee2bb1348d6a36d8ce39df7d3aa, State: Initialized, Role: FOLLOWER I0507 06:32:17.735815 8615 server_base.cc:608] Dumped server information to /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.RestartClusterWithNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data/info.pb I0507 06:32:17.736367 8630 consensus_queue.cc:229] T 3a7d8f424df742599d3f6d48d9b1a37d P 3ce79ee2bb1348d6a36d8ce39df7d3aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3ce79ee2bb1348d6a36d8ce39df7d3aa" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 44481 } } peers { permanent_uuid: "d9fd45507a8e435abc710b00a27538c6" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 44183 } } peers { permanent_uuid: "2781869a927549908dde07bc5019d0e4" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 40501 } } peers { permanent_uuid: "b5c46129f1994ab28f3710e8e392ceab" member_type: NON_VOTER last_known_addr { host: "127.0.57.196" port: 34535 } attrs { promote: true } } I0507 06:32:17.738385 8630 ts_tablet_manager.cc:1081] T 3a7d8f424df742599d3f6d48d9b1a37d P 3ce79ee2bb1348d6a36d8ce39df7d3aa: Time spent starting tablet: real 0.018s user 0.015s sys 0.000s W0507 06:32:17.740378 8736 heartbeater.cc:587] Failed to heartbeat to 127.0.57.254:34231 (0 consecutive failures): Network error: Failed to ping master at 127.0.57.254:34231: Client connection negotiation failed: client connection to 127.0.57.254:34231: connect: Connection refused (error 111) I0507 06:32:17.741044 231 external_mini_cluster.cc:995] Started /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver as pid 8615 I0507 06:32:17.741629 231 external_mini_cluster.cc:933] Running /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.RestartClusterWithNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-1/wal --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.RestartClusterWithNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-1/data --block_manager=log --rpc_bind_addresses=127.0.57.194:44183 --local_ip_for_outbound_sockets=127.0.57.194 --webserver_port=35977 --webserver_interface=127.0.57.194 --tserver_master_addrs=127.0.57.254:34231 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.RestartClusterWithNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-1/logs --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.RestartClusterWithNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-1/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=true --tablet_copy_download_file_inject_latency_ms=10000 --follower_unavailable_considered_failed_sec=2 --consensus_rpc_timeout_ms=5000 with env {} W0507 06:32:17.742076 8736 heartbeater.cc:380] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts. WARNING: Logging before InitGoogleLogging() is written to STDERR W0507 06:32:17.916766 8740 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0507 06:32:17.917023 8740 flags.cc:404] Enabled unsafe flag: --never_fsync=true W0507 06:32:17.919191 8740 flags.cc:404] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true W0507 06:32:17.919245 8740 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W0507 06:32:17.919374 8740 flags.cc:404] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.57.194 I0507 06:32:17.923694 8740 tablet_server_main.cc:78] Tablet server non-default flags: --consensus_rpc_timeout_ms=5000 --follower_unavailable_considered_failed_sec=2 --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.RestartClusterWithNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-1/data --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.RestartClusterWithNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-1/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.0.57.194:44183 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.RestartClusterWithNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-1/data/info.pb --webserver_interface=127.0.57.194 --webserver_port=35977 --tablet_copy_download_file_inject_latency_ms=10000 --tserver_master_addrs=127.0.57.254:34231 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.0.57.194 --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.RestartClusterWithNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-1/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.10.0-cdh6.x-SNAPSHOT revision 7d0ee26697926e87af9ef395008799577f6ab43f build type FASTDEBUG built by jenkins at 06 May 2019 22:44:21 PST on quasar-knmpfx-1.vpc.cloudera.com build id 322017 ASAN enabled I0507 06:32:17.931798 8740 system_ntp.cc:202] NTP initialized. Skew: 500ppm Current error: 143016us I0507 06:32:17.932570 8740 fs_manager.cc:263] Metadata directory not provided I0507 06:32:17.932658 8740 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0507 06:32:17.938832 8740 fs_manager.cc:399] Time spent opening directory manager: real 0.005s user 0.003s sys 0.001s I0507 06:32:17.939525 8740 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0507 06:32:17.940201 8740 file_cache.cc:466] Constructed file cache lbm with capacity 419430 I0507 06:32:17.943290 8740 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s I0507 06:32:17.943393 8740 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.RestartClusterWithNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.RestartClusterWithNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-1/wal uuid: "d9fd45507a8e435abc710b00a27538c6" format_stamp: "Formatted at 2019-05-07 06:32:01 on dist-test-slave-dist-test-slave-rqp6" I0507 06:32:17.944445 8740 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.RestartClusterWithNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-1/wal metadata directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.RestartClusterWithNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-1/wal 1 data directories: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.RestartClusterWithNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-1/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0507 06:32:18.025404 8740 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0507 06:32:18.027595 8740 ts_tablet_manager.cc:344] Loading tablet metadata (0/1 complete) I0507 06:32:18.031905 8740 ts_tablet_manager.cc:356] Loaded tablet metadata (1 total tablets, 1 live tablets) I0507 06:32:18.031970 8740 ts_tablet_manager.cc:362] Registering tablets (0/1 complete) I0507 06:32:18.035197 8755 tablet_bootstrap.cc:439] T 3a7d8f424df742599d3f6d48d9b1a37d P d9fd45507a8e435abc710b00a27538c6: Bootstrap starting. I0507 06:32:18.044654 8740 ts_tablet_manager.cc:376] Registered 1 tablets I0507 06:32:18.055764 8755 log.cc:525] T 3a7d8f424df742599d3f6d48d9b1a37d P d9fd45507a8e435abc710b00a27538c6: Log is configured to *not* fsync() on all Append() calls I0507 06:32:18.089637 8755 tablet_bootstrap.cc:439] T 3a7d8f424df742599d3f6d48d9b1a37d P d9fd45507a8e435abc710b00a27538c6: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=300 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I0507 06:32:18.090565 8755 tablet_bootstrap.cc:439] T 3a7d8f424df742599d3f6d48d9b1a37d P d9fd45507a8e435abc710b00a27538c6: Bootstrap complete. I0507 06:32:18.092034 8755 ts_tablet_manager.cc:1053] T 3a7d8f424df742599d3f6d48d9b1a37d P d9fd45507a8e435abc710b00a27538c6: Time spent bootstrapping tablet: real 0.057s user 0.035s sys 0.014s I0507 06:32:18.093137 8740 rpc_server.cc:205] RPC server started. Bound to: 127.0.57.194:44183 I0507 06:32:18.094557 8740 webserver.cc:308] Webserver started at http://127.0.57.194:35977/ using document root and password file I0507 06:32:18.095268 8740 server_base.cc:608] Dumped server information to /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.RestartClusterWithNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-1/data/info.pb W0507 06:32:18.100236 8861 heartbeater.cc:587] Failed to heartbeat to 127.0.57.254:34231 (0 consecutive failures): Network error: Failed to ping master at 127.0.57.254:34231: Client connection negotiation failed: client connection to 127.0.57.254:34231: connect: Connection refused (error 111) I0507 06:32:18.101451 231 external_mini_cluster.cc:995] Started /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver as pid 8740 I0507 06:32:18.101915 231 external_mini_cluster.cc:933] Running /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.RestartClusterWithNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-3/wal --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.RestartClusterWithNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-3/data --block_manager=log --rpc_bind_addresses=127.0.57.196:34535 --local_ip_for_outbound_sockets=127.0.57.196 --webserver_port=35825 --webserver_interface=127.0.57.196 --tserver_master_addrs=127.0.57.254:34231 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.RestartClusterWithNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-3/logs --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.RestartClusterWithNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-3/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=true --tablet_copy_download_file_inject_latency_ms=10000 --follower_unavailable_considered_failed_sec=2 --consensus_rpc_timeout_ms=5000 with env {} W0507 06:32:18.102074 8861 heartbeater.cc:380] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts. I0507 06:32:18.103062 8755 raft_consensus.cc:340] T 3a7d8f424df742599d3f6d48d9b1a37d P d9fd45507a8e435abc710b00a27538c6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3ce79ee2bb1348d6a36d8ce39df7d3aa" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 44481 } } peers { permanent_uuid: "d9fd45507a8e435abc710b00a27538c6" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 44183 } } peers { permanent_uuid: "2781869a927549908dde07bc5019d0e4" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 40501 } } peers { permanent_uuid: "b5c46129f1994ab28f3710e8e392ceab" member_type: NON_VOTER last_known_addr { host: "127.0.57.196" port: 34535 } attrs { promote: true } } I0507 06:32:18.105746 8755 raft_consensus.cc:705] T 3a7d8f424df742599d3f6d48d9b1a37d P d9fd45507a8e435abc710b00a27538c6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9fd45507a8e435abc710b00a27538c6, State: Initialized, Role: FOLLOWER I0507 06:32:18.106308 8755 consensus_queue.cc:229] T 3a7d8f424df742599d3f6d48d9b1a37d P d9fd45507a8e435abc710b00a27538c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3ce79ee2bb1348d6a36d8ce39df7d3aa" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 44481 } } peers { permanent_uuid: "d9fd45507a8e435abc710b00a27538c6" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 44183 } } peers { permanent_uuid: "2781869a927549908dde07bc5019d0e4" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 40501 } } peers { permanent_uuid: "b5c46129f1994ab28f3710e8e392ceab" member_type: NON_VOTER last_known_addr { host: "127.0.57.196" port: 34535 } attrs { promote: true } } I0507 06:32:18.107812 8755 ts_tablet_manager.cc:1081] T 3a7d8f424df742599d3f6d48d9b1a37d P d9fd45507a8e435abc710b00a27538c6: Time spent starting tablet: real 0.016s user 0.015s sys 0.000s WARNING: Logging before InitGoogleLogging() is written to STDERR W0507 06:32:18.265683 8865 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0507 06:32:18.265895 8865 flags.cc:404] Enabled unsafe flag: --never_fsync=true W0507 06:32:18.267926 8865 flags.cc:404] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true W0507 06:32:18.267992 8865 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W0507 06:32:18.268085 8865 flags.cc:404] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.57.196 I0507 06:32:18.272514 8865 tablet_server_main.cc:78] Tablet server non-default flags: --consensus_rpc_timeout_ms=5000 --follower_unavailable_considered_failed_sec=2 --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.RestartClusterWithNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-3/data --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.RestartClusterWithNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-3/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.0.57.196:34535 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.RestartClusterWithNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-3/data/info.pb --webserver_interface=127.0.57.196 --webserver_port=35825 --tablet_copy_download_file_inject_latency_ms=10000 --tserver_master_addrs=127.0.57.254:34231 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.0.57.196 --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.RestartClusterWithNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-3/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.10.0-cdh6.x-SNAPSHOT revision 7d0ee26697926e87af9ef395008799577f6ab43f build type FASTDEBUG built by jenkins at 06 May 2019 22:44:21 PST on quasar-knmpfx-1.vpc.cloudera.com build id 322017 ASAN enabled I0507 06:32:18.279655 8865 system_ntp.cc:202] NTP initialized. Skew: 500ppm Current error: 143516us I0507 06:32:18.280294 8865 fs_manager.cc:263] Metadata directory not provided I0507 06:32:18.280372 8865 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0507 06:32:18.286088 8865 fs_manager.cc:399] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s I0507 06:32:18.286743 8865 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0507 06:32:18.287294 8865 file_cache.cc:466] Constructed file cache lbm with capacity 419430 I0507 06:32:18.290493 8865 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s I0507 06:32:18.290602 8865 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.RestartClusterWithNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.RestartClusterWithNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-3/wal uuid: "b5c46129f1994ab28f3710e8e392ceab" format_stamp: "Formatted at 2019-05-07 06:32:02 on dist-test-slave-dist-test-slave-rqp6" I0507 06:32:18.291551 8865 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.RestartClusterWithNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-3/wal metadata directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.RestartClusterWithNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-3/wal 1 data directories: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.RestartClusterWithNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-3/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0507 06:32:18.311609 8865 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0507 06:32:18.313300 8865 ts_tablet_manager.cc:344] Loading tablet metadata (0/1 complete) I0507 06:32:18.316931 8865 ts_tablet_manager.cc:356] Loaded tablet metadata (1 total tablets, 1 live tablets) I0507 06:32:18.316987 8865 ts_tablet_manager.cc:362] Registering tablets (0/1 complete) I0507 06:32:18.319448 8865 ts_tablet_manager.cc:376] Registered 1 tablets I0507 06:32:18.320066 8880 tablet_bootstrap.cc:439] T 3a7d8f424df742599d3f6d48d9b1a37d P b5c46129f1994ab28f3710e8e392ceab: Bootstrap starting. I0507 06:32:18.341392 8880 log.cc:525] T 3a7d8f424df742599d3f6d48d9b1a37d P b5c46129f1994ab28f3710e8e392ceab: Log is configured to *not* fsync() on all Append() calls I0507 06:32:18.359666 8865 rpc_server.cc:205] RPC server started. Bound to: 127.0.57.196:34535 I0507 06:32:18.360867 8865 webserver.cc:308] Webserver started at http://127.0.57.196:35825/ using document root and password file I0507 06:32:18.361521 8865 server_base.cc:608] Dumped server information to /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.RestartClusterWithNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-3/data/info.pb I0507 06:32:18.362640 231 external_mini_cluster.cc:995] Started /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver as pid 8865 I0507 06:32:18.363217 231 external_mini_cluster.cc:933] Running /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.RestartClusterWithNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-4/wal --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.RestartClusterWithNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-4/data --block_manager=log --rpc_bind_addresses=127.0.57.197:44221 --local_ip_for_outbound_sockets=127.0.57.197 --webserver_port=41379 --webserver_interface=127.0.57.197 --tserver_master_addrs=127.0.57.254:34231 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.RestartClusterWithNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-4/logs --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.RestartClusterWithNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-4/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=true --tablet_copy_download_file_inject_latency_ms=10000 --follower_unavailable_considered_failed_sec=2 --consensus_rpc_timeout_ms=5000 with env {} W0507 06:32:18.371332 8986 heartbeater.cc:587] Failed to heartbeat to 127.0.57.254:34231 (0 consecutive failures): Network error: Failed to ping master at 127.0.57.254:34231: Client connection negotiation failed: client connection to 127.0.57.254:34231: connect: Connection refused (error 111) I0507 06:32:18.376605 8880 tablet_bootstrap.cc:439] T 3a7d8f424df742599d3f6d48d9b1a37d P b5c46129f1994ab28f3710e8e392ceab: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=300 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates W0507 06:32:18.376827 8986 heartbeater.cc:380] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts. I0507 06:32:18.377140 8880 tablet_bootstrap.cc:439] T 3a7d8f424df742599d3f6d48d9b1a37d P b5c46129f1994ab28f3710e8e392ceab: Bootstrap complete. I0507 06:32:18.378072 8880 ts_tablet_manager.cc:1053] T 3a7d8f424df742599d3f6d48d9b1a37d P b5c46129f1994ab28f3710e8e392ceab: Time spent bootstrapping tablet: real 0.058s user 0.030s sys 0.020s I0507 06:32:18.394052 8880 raft_consensus.cc:340] T 3a7d8f424df742599d3f6d48d9b1a37d P b5c46129f1994ab28f3710e8e392ceab [term 1 LEARNER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3ce79ee2bb1348d6a36d8ce39df7d3aa" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 44481 } } peers { permanent_uuid: "d9fd45507a8e435abc710b00a27538c6" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 44183 } } peers { permanent_uuid: "2781869a927549908dde07bc5019d0e4" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 40501 } } peers { permanent_uuid: "b5c46129f1994ab28f3710e8e392ceab" member_type: NON_VOTER last_known_addr { host: "127.0.57.196" port: 34535 } attrs { promote: true } } I0507 06:32:18.396458 8880 raft_consensus.cc:705] T 3a7d8f424df742599d3f6d48d9b1a37d P b5c46129f1994ab28f3710e8e392ceab [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b5c46129f1994ab28f3710e8e392ceab, State: Initialized, Role: LEARNER I0507 06:32:18.400987 8880 consensus_queue.cc:229] T 3a7d8f424df742599d3f6d48d9b1a37d P b5c46129f1994ab28f3710e8e392ceab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3ce79ee2bb1348d6a36d8ce39df7d3aa" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 44481 } } peers { permanent_uuid: "d9fd45507a8e435abc710b00a27538c6" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 44183 } } peers { permanent_uuid: "2781869a927549908dde07bc5019d0e4" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 40501 } } peers { permanent_uuid: "b5c46129f1994ab28f3710e8e392ceab" member_type: NON_VOTER last_known_addr { host: "127.0.57.196" port: 34535 } attrs { promote: true } } I0507 06:32:18.405282 8880 ts_tablet_manager.cc:1081] T 3a7d8f424df742599d3f6d48d9b1a37d P b5c46129f1994ab28f3710e8e392ceab: Time spent starting tablet: real 0.027s user 0.007s sys 0.006s WARNING: Logging before InitGoogleLogging() is written to STDERR W0507 06:32:18.550235 8989 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0507 06:32:18.550492 8989 flags.cc:404] Enabled unsafe flag: --never_fsync=true W0507 06:32:18.552641 8989 flags.cc:404] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true W0507 06:32:18.552698 8989 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W0507 06:32:18.552800 8989 flags.cc:404] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.57.197 I0507 06:32:18.556927 8989 tablet_server_main.cc:78] Tablet server non-default flags: --consensus_rpc_timeout_ms=5000 --follower_unavailable_considered_failed_sec=2 --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.RestartClusterWithNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-4/data --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.RestartClusterWithNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-4/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.0.57.197:44221 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.RestartClusterWithNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-4/data/info.pb --webserver_interface=127.0.57.197 --webserver_port=41379 --tablet_copy_download_file_inject_latency_ms=10000 --tserver_master_addrs=127.0.57.254:34231 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.0.57.197 --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.RestartClusterWithNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-4/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.10.0-cdh6.x-SNAPSHOT revision 7d0ee26697926e87af9ef395008799577f6ab43f build type FASTDEBUG built by jenkins at 06 May 2019 22:44:21 PST on quasar-knmpfx-1.vpc.cloudera.com build id 322017 ASAN enabled I0507 06:32:18.565032 8989 system_ntp.cc:202] NTP initialized. Skew: 500ppm Current error: 143516us I0507 06:32:18.565781 8989 fs_manager.cc:263] Metadata directory not provided I0507 06:32:18.565868 8989 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0507 06:32:18.572166 8989 fs_manager.cc:399] Time spent opening directory manager: real 0.005s user 0.005s sys 0.000s I0507 06:32:18.572763 8989 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0507 06:32:18.573375 8989 file_cache.cc:466] Constructed file cache lbm with capacity 419430 I0507 06:32:18.577085 8989 fs_manager.cc:419] Time spent opening block manager: real 0.003s user 0.000s sys 0.000s I0507 06:32:18.577191 8989 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.RestartClusterWithNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.RestartClusterWithNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-4/wal uuid: "bb5a49b22d4048608d9e7b762d749fb2" format_stamp: "Formatted at 2019-05-07 06:32:02 on dist-test-slave-dist-test-slave-rqp6" I0507 06:32:18.578233 8989 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.RestartClusterWithNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-4/wal metadata directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.RestartClusterWithNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-4/wal 1 data directories: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.RestartClusterWithNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-4/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0507 06:32:18.618521 8989 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0507 06:32:18.620424 8989 ts_tablet_manager.cc:356] Loaded tablet metadata (0 total tablets, 0 live tablets) I0507 06:32:18.620508 8989 ts_tablet_manager.cc:376] Registered 0 tablets I0507 06:32:18.645573 8989 rpc_server.cc:205] RPC server started. Bound to: 127.0.57.197:44221 I0507 06:32:18.646790 8989 webserver.cc:308] Webserver started at http://127.0.57.197:41379/ using document root and password file I0507 06:32:18.647620 8989 server_base.cc:608] Dumped server information to /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.RestartClusterWithNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-4/data/info.pb I0507 06:32:18.648216 231 external_mini_cluster.cc:995] Started /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver as pid 8989 I0507 06:32:18.648805 231 external_mini_cluster.cc:933] Running /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.RestartClusterWithNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-5/wal --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.RestartClusterWithNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-5/data --block_manager=log --rpc_bind_addresses=127.0.57.198:42585 --local_ip_for_outbound_sockets=127.0.57.198 --webserver_port=34145 --webserver_interface=127.0.57.198 --tserver_master_addrs=127.0.57.254:34231 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.RestartClusterWithNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-5/logs --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.RestartClusterWithNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-5/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=true --tablet_copy_download_file_inject_latency_ms=10000 --follower_unavailable_considered_failed_sec=2 --consensus_rpc_timeout_ms=5000 with env {} W0507 06:32:18.657186 9109 heartbeater.cc:587] Failed to heartbeat to 127.0.57.254:34231 (0 consecutive failures): Network error: Failed to ping master at 127.0.57.254:34231: Client connection negotiation failed: client connection to 127.0.57.254:34231: connect: Connection refused (error 111) W0507 06:32:18.658440 9109 heartbeater.cc:380] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts. WARNING: Logging before InitGoogleLogging() is written to STDERR W0507 06:32:18.820132 9112 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0507 06:32:18.820389 9112 flags.cc:404] Enabled unsafe flag: --never_fsync=true W0507 06:32:18.822531 9112 flags.cc:404] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true W0507 06:32:18.822593 9112 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W0507 06:32:18.822700 9112 flags.cc:404] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.57.198 I0507 06:32:18.826997 9112 tablet_server_main.cc:78] Tablet server non-default flags: --consensus_rpc_timeout_ms=5000 --follower_unavailable_considered_failed_sec=2 --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.RestartClusterWithNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-5/data --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.RestartClusterWithNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-5/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.0.57.198:42585 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.RestartClusterWithNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-5/data/info.pb --webserver_interface=127.0.57.198 --webserver_port=34145 --tablet_copy_download_file_inject_latency_ms=10000 --tserver_master_addrs=127.0.57.254:34231 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.0.57.198 --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.RestartClusterWithNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-5/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.10.0-cdh6.x-SNAPSHOT revision 7d0ee26697926e87af9ef395008799577f6ab43f build type FASTDEBUG built by jenkins at 06 May 2019 22:44:21 PST on quasar-knmpfx-1.vpc.cloudera.com build id 322017 ASAN enabled I0507 06:32:18.835618 9112 system_ntp.cc:202] NTP initialized. Skew: 500ppm Current error: 143516us I0507 06:32:18.836380 9112 fs_manager.cc:263] Metadata directory not provided I0507 06:32:18.836459 9112 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0507 06:32:18.842732 9112 fs_manager.cc:399] Time spent opening directory manager: real 0.005s user 0.002s sys 0.002s I0507 06:32:18.843336 9112 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0507 06:32:18.843992 9112 file_cache.cc:466] Constructed file cache lbm with capacity 419430 I0507 06:32:18.847182 9112 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s I0507 06:32:18.847291 9112 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.RestartClusterWithNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.RestartClusterWithNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-5/wal uuid: "3e033fe6ec6f4fe595bdcd8a95a9f4e8" format_stamp: "Formatted at 2019-05-07 06:32:03 on dist-test-slave-dist-test-slave-rqp6" I0507 06:32:18.848361 9112 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.RestartClusterWithNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-5/wal metadata directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.RestartClusterWithNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-5/wal 1 data directories: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.RestartClusterWithNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-5/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0507 06:32:18.889071 9112 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0507 06:32:18.890976 9112 ts_tablet_manager.cc:356] Loaded tablet metadata (0 total tablets, 0 live tablets) I0507 06:32:18.891060 9112 ts_tablet_manager.cc:376] Registered 0 tablets I0507 06:32:18.919504 9112 rpc_server.cc:205] RPC server started. Bound to: 127.0.57.198:42585 I0507 06:32:18.920773 9112 webserver.cc:308] Webserver started at http://127.0.57.198:34145/ using document root and password file I0507 06:32:18.921711 9112 server_base.cc:608] Dumped server information to /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.RestartClusterWithNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-5/data/info.pb W0507 06:32:18.926440 9232 heartbeater.cc:587] Failed to heartbeat to 127.0.57.254:34231 (0 consecutive failures): Network error: Failed to ping master at 127.0.57.254:34231: Client connection negotiation failed: client connection to 127.0.57.254:34231: connect: Connection refused (error 111) W0507 06:32:18.928306 9232 heartbeater.cc:380] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts. I0507 06:32:18.928515 231 external_mini_cluster.cc:995] Started /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver as pid 9112 I0507 06:32:18.929067 231 external_mini_cluster.cc:933] Running /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.RestartClusterWithNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-6/wal --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.RestartClusterWithNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-6/data --block_manager=log --rpc_bind_addresses=127.0.57.199:44671 --local_ip_for_outbound_sockets=127.0.57.199 --webserver_port=42325 --webserver_interface=127.0.57.199 --tserver_master_addrs=127.0.57.254:34231 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.RestartClusterWithNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-6/logs --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.RestartClusterWithNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-6/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=true --tablet_copy_download_file_inject_latency_ms=10000 --follower_unavailable_considered_failed_sec=2 --consensus_rpc_timeout_ms=5000 with env {} WARNING: Logging before InitGoogleLogging() is written to STDERR W0507 06:32:19.112356 9235 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0507 06:32:19.112887 9235 flags.cc:404] Enabled unsafe flag: --never_fsync=true W0507 06:32:19.115288 9235 flags.cc:404] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true W0507 06:32:19.115355 9235 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W0507 06:32:19.115576 9235 flags.cc:404] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.57.199 I0507 06:32:19.120887 9235 tablet_server_main.cc:78] Tablet server non-default flags: --consensus_rpc_timeout_ms=5000 --follower_unavailable_considered_failed_sec=2 --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.RestartClusterWithNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-6/data --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.RestartClusterWithNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-6/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.0.57.199:44671 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.RestartClusterWithNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-6/data/info.pb --webserver_interface=127.0.57.199 --webserver_port=42325 --tablet_copy_download_file_inject_latency_ms=10000 --tserver_master_addrs=127.0.57.254:34231 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.0.57.199 --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.RestartClusterWithNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-6/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.10.0-cdh6.x-SNAPSHOT revision 7d0ee26697926e87af9ef395008799577f6ab43f build type FASTDEBUG built by jenkins at 06 May 2019 22:44:21 PST on quasar-knmpfx-1.vpc.cloudera.com build id 322017 ASAN enabled I0507 06:32:19.130630 9235 system_ntp.cc:202] NTP initialized. Skew: 500ppm Current error: 144016us I0507 06:32:19.131536 9235 fs_manager.cc:263] Metadata directory not provided I0507 06:32:19.131628 9235 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0507 06:32:19.138886 9235 fs_manager.cc:399] Time spent opening directory manager: real 0.005s user 0.002s sys 0.002s I0507 06:32:19.139691 9235 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0507 06:32:19.140605 9235 file_cache.cc:466] Constructed file cache lbm with capacity 419430 I0507 06:32:19.144223 9235 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s I0507 06:32:19.144330 9235 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.RestartClusterWithNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.RestartClusterWithNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-6/wal uuid: "44a1757d4e604357a1991e808f009e56" format_stamp: "Formatted at 2019-05-07 06:32:03 on dist-test-slave-dist-test-slave-rqp6" I0507 06:32:19.145565 9235 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.RestartClusterWithNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-6/wal metadata directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.RestartClusterWithNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-6/wal 1 data directories: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.RestartClusterWithNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-6/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0507 06:32:19.210636 9235 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0507 06:32:19.212369 9235 ts_tablet_manager.cc:356] Loaded tablet metadata (0 total tablets, 0 live tablets) I0507 06:32:19.212443 9235 ts_tablet_manager.cc:376] Registered 0 tablets I0507 06:32:19.239151 9235 rpc_server.cc:205] RPC server started. Bound to: 127.0.57.199:44671 I0507 06:32:19.240381 9235 webserver.cc:308] Webserver started at http://127.0.57.199:42325/ using document root and password file I0507 06:32:19.241168 9235 server_base.cc:608] Dumped server information to /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.RestartClusterWithNonVoter.1557210500371863-231/raft_consensus-itest-cluster/ts-6/data/info.pb W0507 06:32:19.246080 9355 heartbeater.cc:587] Failed to heartbeat to 127.0.57.254:34231 (0 consecutive failures): Network error: Failed to ping master at 127.0.57.254:34231: Client connection negotiation failed: client connection to 127.0.57.254:34231: connect: Connection refused (error 111) W0507 06:32:19.247500 9355 heartbeater.cc:380] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts. I0507 06:32:19.249521 231 external_mini_cluster.cc:995] Started /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver as pid 9235 I0507 06:32:19.250139 231 external_mini_cluster.cc:933] Running /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-master /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-master --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.RestartClusterWithNonVoter.1557210500371863-231/raft_consensus-itest-cluster/master-0/wal --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.RestartClusterWithNonVoter.1557210500371863-231/raft_consensus-itest-cluster/master-0/data --block_manager=log --webserver_interface=localhost --ipki_ca_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.0.57.254:34231 --webserver_interface=127.0.57.254 --webserver_port=33459 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.RestartClusterWithNonVoter.1557210500371863-231/raft_consensus-itest-cluster/master-0/logs --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.RestartClusterWithNonVoter.1557210500371863-231/raft_consensus-itest-cluster/master-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=true --rpc_reuseport=true with env {} I0507 06:32:19.380945 9359 raft_consensus.cc:472] T 3a7d8f424df742599d3f6d48d9b1a37d P 3ce79ee2bb1348d6a36d8ce39df7d3aa [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0507 06:32:19.381147 9359 raft_consensus.cc:494] T 3a7d8f424df742599d3f6d48d9b1a37d P 3ce79ee2bb1348d6a36d8ce39df7d3aa [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3ce79ee2bb1348d6a36d8ce39df7d3aa" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 44481 } } peers { permanent_uuid: "d9fd45507a8e435abc710b00a27538c6" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 44183 } } peers { permanent_uuid: "2781869a927549908dde07bc5019d0e4" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 40501 } } peers { permanent_uuid: "b5c46129f1994ab28f3710e8e392ceab" member_type: NON_VOTER last_known_addr { host: "127.0.57.196" port: 34535 } attrs { promote: true } } I0507 06:32:19.382378 9359 leader_election.cc:296] T 3a7d8f424df742599d3f6d48d9b1a37d P 3ce79ee2bb1348d6a36d8ce39df7d3aa [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d9fd45507a8e435abc710b00a27538c6 (127.0.57.194:44183), 2781869a927549908dde07bc5019d0e4 (127.0.57.195:40501) W0507 06:32:19.390130 8624 leader_election.cc:341] T 3a7d8f424df742599d3f6d48d9b1a37d P 3ce79ee2bb1348d6a36d8ce39df7d3aa [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2781869a927549908dde07bc5019d0e4 (127.0.57.195:40501): Network error: Client connection negotiation failed: client connection to 127.0.57.195:40501: connect: Connection refused (error 111) I0507 06:32:19.397958 8818 tablet_service.cc:1286] Received RequestConsensusVote() RPC: tablet_id: "3a7d8f424df742599d3f6d48d9b1a37d" candidate_uuid: "3ce79ee2bb1348d6a36d8ce39df7d3aa" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "d9fd45507a8e435abc710b00a27538c6" is_pre_election: true I0507 06:32:19.398362 8818 raft_consensus.cc:2346] T 3a7d8f424df742599d3f6d48d9b1a37d P d9fd45507a8e435abc710b00a27538c6 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3ce79ee2bb1348d6a36d8ce39df7d3aa in term 1. I0507 06:32:19.399374 8623 leader_election.cc:310] T 3a7d8f424df742599d3f6d48d9b1a37d P 3ce79ee2bb1348d6a36d8ce39df7d3aa [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3ce79ee2bb1348d6a36d8ce39df7d3aa, d9fd45507a8e435abc710b00a27538c6; no voters: 2781869a927549908dde07bc5019d0e4 I0507 06:32:19.399883 9359 raft_consensus.cc:2640] T 3a7d8f424df742599d3f6d48d9b1a37d P 3ce79ee2bb1348d6a36d8ce39df7d3aa [term 1 FOLLOWER]: Leader pre-election won for term 2 I0507 06:32:19.399987 9359 raft_consensus.cc:472] T 3a7d8f424df742599d3f6d48d9b1a37d P 3ce79ee2bb1348d6a36d8ce39df7d3aa [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0507 06:32:19.400051 9359 raft_consensus.cc:2886] T 3a7d8f424df742599d3f6d48d9b1a37d P 3ce79ee2bb1348d6a36d8ce39df7d3aa [term 1 FOLLOWER]: Advancing to term 2 I0507 06:32:19.403178 9359 raft_consensus.cc:494] T 3a7d8f424df742599d3f6d48d9b1a37d P 3ce79ee2bb1348d6a36d8ce39df7d3aa [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3ce79ee2bb1348d6a36d8ce39df7d3aa" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 44481 } } peers { permanent_uuid: "d9fd45507a8e435abc710b00a27538c6" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 44183 } } peers { permanent_uuid: "2781869a927549908dde07bc5019d0e4" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 40501 } } peers { permanent_uuid: "b5c46129f1994ab28f3710e8e392ceab" member_type: NON_VOTER last_known_addr { host: "127.0.57.196" port: 34535 } attrs { promote: true } } I0507 06:32:19.403795 9359 leader_election.cc:296] T 3a7d8f424df742599d3f6d48d9b1a37d P 3ce79ee2bb1348d6a36d8ce39df7d3aa [CANDIDATE]: Term 2 election: Requested vote from peers d9fd45507a8e435abc710b00a27538c6 (127.0.57.194:44183), 2781869a927549908dde07bc5019d0e4 (127.0.57.195:40501) W0507 06:32:19.404436 8624 leader_election.cc:341] T 3a7d8f424df742599d3f6d48d9b1a37d P 3ce79ee2bb1348d6a36d8ce39df7d3aa [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 2781869a927549908dde07bc5019d0e4 (127.0.57.195:40501): Network error: Client connection negotiation failed: client connection to 127.0.57.195:40501: connect: Connection refused (error 111) I0507 06:32:19.404373 8818 tablet_service.cc:1286] Received RequestConsensusVote() RPC: tablet_id: "3a7d8f424df742599d3f6d48d9b1a37d" candidate_uuid: "3ce79ee2bb1348d6a36d8ce39df7d3aa" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "d9fd45507a8e435abc710b00a27538c6" I0507 06:32:19.404587 8818 raft_consensus.cc:2886] T 3a7d8f424df742599d3f6d48d9b1a37d P d9fd45507a8e435abc710b00a27538c6 [term 1 FOLLOWER]: Advancing to term 2 I0507 06:32:19.407413 8818 raft_consensus.cc:2346] T 3a7d8f424df742599d3f6d48d9b1a37d P d9fd45507a8e435abc710b00a27538c6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ce79ee2bb1348d6a36d8ce39df7d3aa in term 2. I0507 06:32:19.407841 8623 leader_election.cc:310] T 3a7d8f424df742599d3f6d48d9b1a37d P 3ce79ee2bb1348d6a36d8ce39df7d3aa [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3ce79ee2bb1348d6a36d8ce39df7d3aa, d9fd45507a8e435abc710b00a27538c6; no voters: 2781869a927549908dde07bc5019d0e4 I0507 06:32:19.408226 9359 raft_consensus.cc:2640] T 3a7d8f424df742599d3f6d48d9b1a37d P 3ce79ee2bb1348d6a36d8ce39df7d3aa [term 2 FOLLOWER]: Leader election won for term 2 I0507 06:32:19.412267 9359 raft_consensus.cc:667] T 3a7d8f424df742599d3f6d48d9b1a37d P 3ce79ee2bb1348d6a36d8ce39df7d3aa [term 2 LEADER]: Becoming Leader. State: Replica: 3ce79ee2bb1348d6a36d8ce39df7d3aa, State: Running, Role: LEADER I0507 06:32:19.412736 9359 consensus_queue.cc:206] T 3a7d8f424df742599d3f6d48d9b1a37d P 3ce79ee2bb1348d6a36d8ce39df7d3aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3ce79ee2bb1348d6a36d8ce39df7d3aa" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 44481 } } peers { permanent_uuid: "d9fd45507a8e435abc710b00a27538c6" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 44183 } } peers { permanent_uuid: "2781869a927549908dde07bc5019d0e4" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 40501 } } peers { permanent_uuid: "b5c46129f1994ab28f3710e8e392ceab" member_type: NON_VOTER last_known_addr { host: "127.0.57.196" port: 34535 } attrs { promote: true } } WARNING: Logging before InitGoogleLogging() is written to STDERR W0507 06:32:19.451498 9358 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0507 06:32:19.451715 9358 flags.cc:404] Enabled unsafe flag: --never_fsync=true W0507 06:32:19.454216 9358 flags.cc:404] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true W0507 06:32:19.454273 9358 flags.cc:404] Enabled experimental flag: --ipki_ca_key_size=1024 W0507 06:32:19.454306 9358 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W0507 06:32:19.454336 9358 flags.cc:404] Enabled experimental flag: --tsk_num_rsa_bits=512 W0507 06:32:19.454370 9358 flags.cc:404] Enabled experimental flag: --rpc_reuseport=true I0507 06:32:19.459709 9358 master_main.cc:73] Master server non-default flags: --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.RestartClusterWithNonVoter.1557210500371863-231/raft_consensus-itest-cluster/master-0/data --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.RestartClusterWithNonVoter.1557210500371863-231/raft_consensus-itest-cluster/master-0/wal --ipki_ca_key_size=1024 --ipki_server_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.0.57.254:34231 --rpc_reuseport=true --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.RestartClusterWithNonVoter.1557210500371863-231/raft_consensus-itest-cluster/master-0/data/info.pb --webserver_interface=127.0.57.254 --webserver_port=33459 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.RestartClusterWithNonVoter.1557210500371863-231/raft_consensus-itest-cluster/master-0/logs --logbuflevel=-1 --logtostderr=true Master server version: kudu 1.10.0-cdh6.x-SNAPSHOT revision 7d0ee26697926e87af9ef395008799577f6ab43f build type FASTDEBUG built by jenkins at 06 May 2019 22:44:21 PST on quasar-knmpfx-1.vpc.cloudera.com build id 322017 ASAN enabled I0507 06:32:19.466295 9358 system_ntp.cc:202] NTP initialized. Skew: 500ppm Current error: 144016us I0507 06:32:19.467532 9358 fs_manager.cc:263] Metadata directory not provided I0507 06:32:19.467623 9358 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0507 06:32:19.473415 9358 fs_manager.cc:399] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s I0507 06:32:19.474050 9358 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0507 06:32:19.474592 9358 file_cache.cc:466] Constructed file cache lbm with capacity 419430 I0507 06:32:19.478350 9358 fs_manager.cc:419] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s I0507 06:32:19.478452 9358 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.RestartClusterWithNonVoter.1557210500371863-231/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.RestartClusterWithNonVoter.1557210500371863-231/raft_consensus-itest-cluster/master-0/wal uuid: "5fbef3ba10d6481bac975db052d4d399" format_stamp: "Formatted at 2019-05-07 06:32:01 on dist-test-slave-dist-test-slave-rqp6" I0507 06:32:19.479408 9358 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.RestartClusterWithNonVoter.1557210500371863-231/raft_consensus-itest-cluster/master-0/wal metadata directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.RestartClusterWithNonVoter.1557210500371863-231/raft_consensus-itest-cluster/master-0/wal 1 data directories: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.RestartClusterWithNonVoter.1557210500371863-231/raft_consensus-itest-cluster/master-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0507 06:32:19.512040 9358 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0507 06:32:19.526682 9358 rpc_server.cc:205] RPC server started. Bound to: 127.0.57.254:34231 I0507 06:32:19.527770 9358 webserver.cc:308] Webserver started at http://127.0.57.254:33459/ using document root and password file I0507 06:32:19.528594 9358 server_base.cc:608] Dumped server information to /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.RestartClusterWithNonVoter.1557210500371863-231/raft_consensus-itest-cluster/master-0/data/info.pb I0507 06:32:19.530915 231 external_mini_cluster.cc:995] Started /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-master as pid 9358 I0507 06:32:19.541992 9424 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 5fbef3ba10d6481bac975db052d4d399: Bootstrap starting. I0507 06:32:19.557988 9424 log.cc:525] T 00000000000000000000000000000000 P 5fbef3ba10d6481bac975db052d4d399: Log is configured to *not* fsync() on all Append() calls I0507 06:32:19.568064 9424 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 5fbef3ba10d6481bac975db052d4d399: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=4 ignored=0} mutations{seen=3 ignored=0} orphaned_commits=0. Pending: 0 replicates I0507 06:32:19.568583 9424 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 5fbef3ba10d6481bac975db052d4d399: Bootstrap complete. I0507 06:32:19.575557 9424 raft_consensus.cc:340] T 00000000000000000000000000000000 P 5fbef3ba10d6481bac975db052d4d399 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fbef3ba10d6481bac975db052d4d399" member_type: VOTER } I0507 06:32:19.576357 9424 raft_consensus.cc:705] T 00000000000000000000000000000000 P 5fbef3ba10d6481bac975db052d4d399 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fbef3ba10d6481bac975db052d4d399, State: Initialized, Role: FOLLOWER I0507 06:32:19.576912 9424 consensus_queue.cc:229] T 00000000000000000000000000000000 P 5fbef3ba10d6481bac975db052d4d399 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fbef3ba10d6481bac975db052d4d399" member_type: VOTER } I0507 06:32:19.577075 9424 raft_consensus.cc:380] T 00000000000000000000000000000000 P 5fbef3ba10d6481bac975db052d4d399 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0507 06:32:19.577157 9424 raft_consensus.cc:472] T 00000000000000000000000000000000 P 5fbef3ba10d6481bac975db052d4d399 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0507 06:32:19.577250 9424 raft_consensus.cc:2886] T 00000000000000000000000000000000 P 5fbef3ba10d6481bac975db052d4d399 [term 1 FOLLOWER]: Advancing to term 2 I0507 06:32:19.581610 9424 raft_consensus.cc:494] T 00000000000000000000000000000000 P 5fbef3ba10d6481bac975db052d4d399 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fbef3ba10d6481bac975db052d4d399" member_type: VOTER } I0507 06:32:19.582115 9424 leader_election.cc:310] T 00000000000000000000000000000000 P 5fbef3ba10d6481bac975db052d4d399 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5fbef3ba10d6481bac975db052d4d399; no voters: I0507 06:32:19.582868 9424 leader_election.cc:296] T 00000000000000000000000000000000 P 5fbef3ba10d6481bac975db052d4d399 [CANDIDATE]: Term 2 election: Requested vote from peers I0507 06:32:19.583180 9428 raft_consensus.cc:2640] T 00000000000000000000000000000000 P 5fbef3ba10d6481bac975db052d4d399 [term 2 FOLLOWER]: Leader election won for term 2 I0507 06:32:19.584065 9428 raft_consensus.cc:667] T 00000000000000000000000000000000 P 5fbef3ba10d6481bac975db052d4d399 [term 2 LEADER]: Becoming Leader. State: Replica: 5fbef3ba10d6481bac975db052d4d399, State: Running, Role: LEADER I0507 06:32:19.584594 9428 consensus_queue.cc:206] T 00000000000000000000000000000000 P 5fbef3ba10d6481bac975db052d4d399 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fbef3ba10d6481bac975db052d4d399" member_type: VOTER } I0507 06:32:19.586740 9424 sys_catalog.cc:337] T 00000000000000000000000000000000 P 5fbef3ba10d6481bac975db052d4d399 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 2 leader_uuid: "5fbef3ba10d6481bac975db052d4d399" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fbef3ba10d6481bac975db052d4d399" member_type: VOTER } } I0507 06:32:19.586953 9430 sys_catalog.cc:337] T 00000000000000000000000000000000 P 5fbef3ba10d6481bac975db052d4d399 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5fbef3ba10d6481bac975db052d4d399. Latest consensus state: current_term: 2 leader_uuid: "5fbef3ba10d6481bac975db052d4d399" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fbef3ba10d6481bac975db052d4d399" member_type: VOTER } } I0507 06:32:19.587365 9430 sys_catalog.cc:340] T 00000000000000000000000000000000 P 5fbef3ba10d6481bac975db052d4d399 [sys.catalog]: This master's current role is: LEADER I0507 06:32:19.587119 9429 sys_catalog.cc:337] T 00000000000000000000000000000000 P 5fbef3ba10d6481bac975db052d4d399 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "5fbef3ba10d6481bac975db052d4d399" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fbef3ba10d6481bac975db052d4d399" member_type: VOTER } } I0507 06:32:19.587306 9424 sys_catalog.cc:340] T 00000000000000000000000000000000 P 5fbef3ba10d6481bac975db052d4d399 [sys.catalog]: This master's current role is: LEADER I0507 06:32:19.587453 9429 sys_catalog.cc:340] T 00000000000000000000000000000000 P 5fbef3ba10d6481bac975db052d4d399 [sys.catalog]: This master's current role is: LEADER I0507 06:32:19.589179 9424 sys_catalog.cc:424] T 00000000000000000000000000000000 P 5fbef3ba10d6481bac975db052d4d399 [sys.catalog]: configured and running, proceeding with master startup. I0507 06:32:19.589298 9432 catalog_manager.cc:1055] Loading table and tablet metadata into memory... I0507 06:32:19.594200 9432 catalog_manager.cc:356] Loaded metadata for table TestTable [id=4245f699bfa84b548c6ed127e9d69518] I0507 06:32:19.596065 9432 catalog_manager.cc:410] Loaded metadata for tablet 3a7d8f424df742599d3f6d48d9b1a37d (table TestTable [id=4245f699bfa84b548c6ed127e9d69518]) I0507 06:32:19.596271 9432 catalog_manager.cc:1066] Initializing Kudu internal certificate authority... I0507 06:32:19.599341 9432 catalog_manager.cc:1075] Loading token signing keys... I0507 06:32:19.600462 9432 catalog_manager.cc:4303] T 00000000000000000000000000000000 P 5fbef3ba10d6481bac975db052d4d399: Loaded TSK: 0 I0507 06:32:19.675412 9109 heartbeater.cc:345] Connected to a master server at 127.0.57.254:34231 I0507 06:32:19.675593 9109 heartbeater.cc:426] Registering TS with master... I0507 06:32:19.676216 9109 heartbeater.cc:475] Master 127.0.57.254:34231 requested a full tablet report, sending... I0507 06:32:19.678261 9391 ts_manager.cc:159] Registered new tserver with Master: bb5a49b22d4048608d9e7b762d749fb2 (127.0.57.197:44221) I0507 06:32:19.680212 9391 master_service.cc:273] Signed X509 certificate for tserver {username='slave'} at 127.0.57.197:34255 I0507 06:32:19.834720 8943 raft_consensus.cc:2886] T 3a7d8f424df742599d3f6d48d9b1a37d P b5c46129f1994ab28f3710e8e392ceab [term 1 LEARNER]: Advancing to term 2 I0507 06:32:19.838115 8943 raft_consensus.cc:1181] T 3a7d8f424df742599d3f6d48d9b1a37d P b5c46129f1994ab28f3710e8e392ceab [term 2 LEARNER]: Refusing update from remote peer 3ce79ee2bb1348d6a36d8ce39df7d3aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch) I0507 06:32:19.839519 9359 consensus_queue.cc:984] T 3a7d8f424df742599d3f6d48d9b1a37d P 3ce79ee2bb1348d6a36d8ce39df7d3aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5c46129f1994ab28f3710e8e392ceab" member_type: NON_VOTER last_known_addr { host: "127.0.57.196" port: 34535 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s I0507 06:32:19.846650 9364 raft_consensus.cc:974] T 3a7d8f424df742599d3f6d48d9b1a37d P 3ce79ee2bb1348d6a36d8ce39df7d3aa: attempting to promote NON_VOTER b5c46129f1994ab28f3710e8e392ceab to VOTER W0507 06:32:19.846828 9364 raft_consensus.cc:978] T 3a7d8f424df742599d3f6d48d9b1a37d P 3ce79ee2bb1348d6a36d8ce39df7d3aa: Unable to promote non-voter b5c46129f1994ab28f3710e8e392ceab: Illegal state: Leader has not yet committed an operation in its own term I0507 06:32:19.847663 8986 heartbeater.cc:345] Connected to a master server at 127.0.57.254:34231 I0507 06:32:19.847807 8986 heartbeater.cc:426] Registering TS with master... I0507 06:32:19.848421 8986 heartbeater.cc:475] Master 127.0.57.254:34231 requested a full tablet report, sending... I0507 06:32:19.850255 9391 ts_manager.cc:159] Registered new tserver with Master: b5c46129f1994ab28f3710e8e392ceab (127.0.57.196:34535) I0507 06:32:19.851305 9391 catalog_manager.cc:3988] T 3a7d8f424df742599d3f6d48d9b1a37d P b5c46129f1994ab28f3710e8e392ceab reported cstate change: term changed from 1 to 2, leader changed from d9fd45507a8e435abc710b00a27538c6 (127.0.57.194) to 3ce79ee2bb1348d6a36d8ce39df7d3aa (127.0.57.193). New cstate: current_term: 2 leader_uuid: "3ce79ee2bb1348d6a36d8ce39df7d3aa" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3ce79ee2bb1348d6a36d8ce39df7d3aa" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 44481 } } peers { permanent_uuid: "d9fd45507a8e435abc710b00a27538c6" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 44183 } } peers { permanent_uuid: "2781869a927549908dde07bc5019d0e4" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 40501 } } peers { permanent_uuid: "b5c46129f1994ab28f3710e8e392ceab" member_type: NON_VOTER last_known_addr { host: "127.0.57.196" port: 34535 } attrs { promote: true } } } I0507 06:32:19.870360 9391 master_service.cc:273] Signed X509 certificate for tserver {username='slave'} at 127.0.57.196:54225 I0507 06:32:19.938894 9232 heartbeater.cc:345] Connected to a master server at 127.0.57.254:34231 I0507 06:32:19.939059 9232 heartbeater.cc:426] Registering TS with master... I0507 06:32:19.939651 9232 heartbeater.cc:475] Master 127.0.57.254:34231 requested a full tablet report, sending... I0507 06:32:19.940788 9391 ts_manager.cc:159] Registered new tserver with Master: 3e033fe6ec6f4fe595bdcd8a95a9f4e8 (127.0.57.198:42585) I0507 06:32:19.941967 9391 master_service.cc:273] Signed X509 certificate for tserver {username='slave'} at 127.0.57.198:34245 I0507 06:32:19.958755 8818 raft_consensus.cc:1181] T 3a7d8f424df742599d3f6d48d9b1a37d P d9fd45507a8e435abc710b00a27538c6 [term 2 FOLLOWER]: Refusing update from remote peer 3ce79ee2bb1348d6a36d8ce39df7d3aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch) I0507 06:32:19.959555 9359 consensus_queue.cc:984] T 3a7d8f424df742599d3f6d48d9b1a37d P 3ce79ee2bb1348d6a36d8ce39df7d3aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9fd45507a8e435abc710b00a27538c6" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 44183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s I0507 06:32:19.970427 8861 heartbeater.cc:345] Connected to a master server at 127.0.57.254:34231 I0507 06:32:19.970577 8861 heartbeater.cc:426] Registering TS with master... I0507 06:32:19.971217 8861 heartbeater.cc:475] Master 127.0.57.254:34231 requested a full tablet report, sending... I0507 06:32:19.977442 9391 ts_manager.cc:159] Registered new tserver with Master: d9fd45507a8e435abc710b00a27538c6 (127.0.57.194:44183) I0507 06:32:19.978878 9391 master_service.cc:273] Signed X509 certificate for tserver {username='slave'} at 127.0.57.194:59501 I0507 06:32:19.980795 8861 heartbeater.cc:467] Master 127.0.57.254:34231 was elected leader, sending a full tablet report... I0507 06:32:19.981420 8986 heartbeater.cc:467] Master 127.0.57.254:34231 was elected leader, sending a full tablet report... I0507 06:32:19.982367 9359 raft_consensus.cc:974] T 3a7d8f424df742599d3f6d48d9b1a37d P 3ce79ee2bb1348d6a36d8ce39df7d3aa: attempting to promote NON_VOTER b5c46129f1994ab28f3710e8e392ceab to VOTER I0507 06:32:19.983311 9359 consensus_queue.cc:206] T 3a7d8f424df742599d3f6d48d9b1a37d P 3ce79ee2bb1348d6a36d8ce39df7d3aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 8, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "3ce79ee2bb1348d6a36d8ce39df7d3aa" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 44481 } } peers { permanent_uuid: "d9fd45507a8e435abc710b00a27538c6" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 44183 } } peers { permanent_uuid: "2781869a927549908dde07bc5019d0e4" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 40501 } } peers { permanent_uuid: "b5c46129f1994ab28f3710e8e392ceab" member_type: VOTER last_known_addr { host: "127.0.57.196" port: 34535 } attrs { promote: false } } I0507 06:32:19.985004 8736 heartbeater.cc:345] Connected to a master server at 127.0.57.254:34231 I0507 06:32:19.985148 8736 heartbeater.cc:426] Registering TS with master... I0507 06:32:19.985529 8943 raft_consensus.cc:1181] T 3a7d8f424df742599d3f6d48d9b1a37d P b5c46129f1994ab28f3710e8e392ceab [term 2 LEARNER]: Refusing update from remote peer 3ce79ee2bb1348d6a36d8ce39df7d3aa: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch) W0507 06:32:19.985635 8624 consensus_peers.cc:457] T 3a7d8f424df742599d3f6d48d9b1a37d P 3ce79ee2bb1348d6a36d8ce39df7d3aa -> Peer 2781869a927549908dde07bc5019d0e4 (127.0.57.195:40501): Couldn't send request to peer 2781869a927549908dde07bc5019d0e4. Status: Network error: Client connection negotiation failed: client connection to 127.0.57.195:40501: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry. I0507 06:32:19.985767 8736 heartbeater.cc:475] Master 127.0.57.254:34231 requested a full tablet report, sending... I0507 06:32:19.986052 9446 consensus_queue.cc:984] T 3a7d8f424df742599d3f6d48d9b1a37d P 3ce79ee2bb1348d6a36d8ce39df7d3aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5c46129f1994ab28f3710e8e392ceab" member_type: VOTER last_known_addr { host: "127.0.57.196" port: 34535 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s I0507 06:32:19.987476 9390 ts_manager.cc:159] Registered new tserver with Master: 3ce79ee2bb1348d6a36d8ce39df7d3aa (127.0.57.193:44481) I0507 06:32:19.988682 9390 master_service.cc:273] Signed X509 certificate for tserver {username='slave'} at 127.0.57.193:46319 I0507 06:32:19.988734 8818 raft_consensus.cc:1181] T 3a7d8f424df742599d3f6d48d9b1a37d P d9fd45507a8e435abc710b00a27538c6 [term 2 FOLLOWER]: Refusing update from remote peer 3ce79ee2bb1348d6a36d8ce39df7d3aa: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch) I0507 06:32:19.989460 9359 consensus_queue.cc:984] T 3a7d8f424df742599d3f6d48d9b1a37d P 3ce79ee2bb1348d6a36d8ce39df7d3aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9fd45507a8e435abc710b00a27538c6" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 44183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s I0507 06:32:19.990629 8736 heartbeater.cc:467] Master 127.0.57.254:34231 was elected leader, sending a full tablet report... I0507 06:32:19.991631 9364 raft_consensus.cc:2788] T 3a7d8f424df742599d3f6d48d9b1a37d P 3ce79ee2bb1348d6a36d8ce39df7d3aa [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 8 to 10, b5c46129f1994ab28f3710e8e392ceab (127.0.57.196) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "3ce79ee2bb1348d6a36d8ce39df7d3aa" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 44481 } } peers { permanent_uuid: "d9fd45507a8e435abc710b00a27538c6" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 44183 } } peers { permanent_uuid: "2781869a927549908dde07bc5019d0e4" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 40501 } } peers { permanent_uuid: "b5c46129f1994ab28f3710e8e392ceab" member_type: VOTER last_known_addr { host: "127.0.57.196" port: 34535 } attrs { promote: false } } } I0507 06:32:19.995203 8943 raft_consensus.cc:2788] T 3a7d8f424df742599d3f6d48d9b1a37d P b5c46129f1994ab28f3710e8e392ceab [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, b5c46129f1994ab28f3710e8e392ceab (127.0.57.196) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "3ce79ee2bb1348d6a36d8ce39df7d3aa" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 44481 } } peers { permanent_uuid: "d9fd45507a8e435abc710b00a27538c6" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 44183 } } peers { permanent_uuid: "2781869a927549908dde07bc5019d0e4" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 40501 } } peers { permanent_uuid: "b5c46129f1994ab28f3710e8e392ceab" member_type: VOTER last_known_addr { host: "127.0.57.196" port: 34535 } attrs { promote: false } } } I0507 06:32:19.995069 8818 raft_consensus.cc:2788] T 3a7d8f424df742599d3f6d48d9b1a37d P d9fd45507a8e435abc710b00a27538c6 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, b5c46129f1994ab28f3710e8e392ceab (127.0.57.196) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "3ce79ee2bb1348d6a36d8ce39df7d3aa" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 44481 } } peers { permanent_uuid: "d9fd45507a8e435abc710b00a27538c6" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 44183 } } peers { permanent_uuid: "2781869a927549908dde07bc5019d0e4" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 40501 } } peers { permanent_uuid: "b5c46129f1994ab28f3710e8e392ceab" member_type: VOTER last_known_addr { host: "127.0.57.196" port: 34535 } attrs { promote: false } } } I0507 06:32:19.996101 9391 catalog_manager.cc:3988] T 3a7d8f424df742599d3f6d48d9b1a37d P 3ce79ee2bb1348d6a36d8ce39df7d3aa reported cstate change: config changed from index 8 to 10, b5c46129f1994ab28f3710e8e392ceab (127.0.57.196) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "3ce79ee2bb1348d6a36d8ce39df7d3aa" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "3ce79ee2bb1348d6a36d8ce39df7d3aa" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 44481 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d9fd45507a8e435abc710b00a27538c6" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 44183 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2781869a927549908dde07bc5019d0e4" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 40501 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b5c46129f1994ab28f3710e8e392ceab" member_type: VOTER last_known_addr { host: "127.0.57.196" port: 34535 } attrs { promote: false } health_report { overall_health: HEALTHY } } } I0507 06:32:20.000315 9391 catalog_manager.cc:3739] Sending ChangeConfig:REMOVE_PEER on tablet 3a7d8f424df742599d3f6d48d9b1a37d (attempt 1) I0507 06:32:20.003362 9388 catalog_manager.cc:3739] Sending ChangeConfig:REMOVE_PEER on tablet 3a7d8f424df742599d3f6d48d9b1a37d (attempt 1) I0507 06:32:20.009224 8692 consensus_queue.cc:206] T 3a7d8f424df742599d3f6d48d9b1a37d P 3ce79ee2bb1348d6a36d8ce39df7d3aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 8, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "3ce79ee2bb1348d6a36d8ce39df7d3aa" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 44481 } } peers { permanent_uuid: "d9fd45507a8e435abc710b00a27538c6" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 44183 } } peers { permanent_uuid: "b5c46129f1994ab28f3710e8e392ceab" member_type: VOTER last_known_addr { host: "127.0.57.196" port: 34535 } attrs { promote: false } } I0507 06:32:20.011355 8943 raft_consensus.cc:1181] T 3a7d8f424df742599d3f6d48d9b1a37d P b5c46129f1994ab28f3710e8e392ceab [term 2 FOLLOWER]: Refusing update from remote peer 3ce79ee2bb1348d6a36d8ce39df7d3aa: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch) I0507 06:32:20.011869 9359 consensus_queue.cc:984] T 3a7d8f424df742599d3f6d48d9b1a37d P 3ce79ee2bb1348d6a36d8ce39df7d3aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5c46129f1994ab28f3710e8e392ceab" member_type: VOTER last_known_addr { host: "127.0.57.196" port: 34535 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s I0507 06:32:20.011963 9372 catalog_manager.cc:3548] ChangeConfig:REMOVE_PEER RPC for tablet 3a7d8f424df742599d3f6d48d9b1a37d with cas_config_opid_index 10: ChangeConfig:REMOVE_PEER failed with leader 3ce79ee2bb1348d6a36d8ce39df7d3aa (127.0.57.193:44481) due to error UNKNOWN_ERROR; will retry: Illegal state: RaftConfig change currently pending. Only one is allowed at a time. Committed config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "3ce79ee2bb1348d6a36d8ce39df7d3aa" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 44481 } } peers { permanent_uuid: "d9fd45507a8e435abc710b00a27538c6" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 44183 } } peers { permanent_uuid: "2781869a927549908dde07bc5019d0e4" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 40501 } } peers { permanent_uuid: "b5c46129f1994ab28f3710e8e392ceab" member_type: VOTER last_known_addr { host: "127.0.57.196" port: 34535 } attrs { promote: false } }. Pending config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "3ce79ee2bb1348d6a36d8ce39df7d3aa" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 44481 } } peers { permanent_uuid: "d9fd45507a8e435abc710b00a27538c6" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 44183 } } peers { permanent_uuid: "b5c46129f1994ab28f3710e8e392ceab" member_type: VOTER last_known_addr { host: "127.0.57.196" port: 34535 } attrs { promote: false } } I0507 06:32:20.012084 9372 catalog_manager.cc:3168] Scheduling retry of ChangeConfig:REMOVE_PEER RPC for tablet 3a7d8f424df742599d3f6d48d9b1a37d with cas_config_opid_index 10 with a delay of 35 ms (attempt = 1) I0507 06:32:20.012619 8818 raft_consensus.cc:1181] T 3a7d8f424df742599d3f6d48d9b1a37d P d9fd45507a8e435abc710b00a27538c6 [term 2 FOLLOWER]: Refusing update from remote peer 3ce79ee2bb1348d6a36d8ce39df7d3aa: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch) I0507 06:32:20.013056 9359 consensus_queue.cc:984] T 3a7d8f424df742599d3f6d48d9b1a37d P 3ce79ee2bb1348d6a36d8ce39df7d3aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9fd45507a8e435abc710b00a27538c6" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 44183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s I0507 06:32:20.013986 9450 raft_consensus.cc:2788] T 3a7d8f424df742599d3f6d48d9b1a37d P 3ce79ee2bb1348d6a36d8ce39df7d3aa [term 2 LEADER]: Committing config change with OpId 2.11: config changed from index 10 to 11, VOTER 2781869a927549908dde07bc5019d0e4 (127.0.57.195) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "3ce79ee2bb1348d6a36d8ce39df7d3aa" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 44481 } } peers { permanent_uuid: "d9fd45507a8e435abc710b00a27538c6" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 44183 } } peers { permanent_uuid: "b5c46129f1994ab28f3710e8e392ceab" member_type: VOTER last_known_addr { host: "127.0.57.196" port: 34535 } attrs { promote: false } } } I0507 06:32:20.014554 8943 raft_consensus.cc:2788] T 3a7d8f424df742599d3f6d48d9b1a37d P b5c46129f1994ab28f3710e8e392ceab [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, VOTER 2781869a927549908dde07bc5019d0e4 (127.0.57.195) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "3ce79ee2bb1348d6a36d8ce39df7d3aa" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 44481 } } peers { permanent_uuid: "d9fd45507a8e435abc710b00a27538c6" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 44183 } } peers { permanent_uuid: "b5c46129f1994ab28f3710e8e392ceab" member_type: VOTER last_known_addr { host: "127.0.57.196" port: 34535 } attrs { promote: false } } } I0507 06:32:20.015950 8818 raft_consensus.cc:2788] T 3a7d8f424df742599d3f6d48d9b1a37d P d9fd45507a8e435abc710b00a27538c6 [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, VOTER 2781869a927549908dde07bc5019d0e4 (127.0.57.195) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "3ce79ee2bb1348d6a36d8ce39df7d3aa" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 44481 } } peers { permanent_uuid: "d9fd45507a8e435abc710b00a27538c6" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 44183 } } peers { permanent_uuid: "b5c46129f1994ab28f3710e8e392ceab" member_type: VOTER last_known_addr { host: "127.0.57.196" port: 34535 } attrs { promote: false } } } I0507 06:32:20.016747 9372 catalog_manager.cc:3531] ChangeConfig:REMOVE_PEER RPC for tablet 3a7d8f424df742599d3f6d48d9b1a37d with cas_config_opid_index 10: ChangeConfig:REMOVE_PEER succeeded (attempt 1) I0507 06:32:20.017927 9388 catalog_manager.cc:3988] T 3a7d8f424df742599d3f6d48d9b1a37d P 3ce79ee2bb1348d6a36d8ce39df7d3aa reported cstate change: config changed from index 10 to 11, VOTER 2781869a927549908dde07bc5019d0e4 (127.0.57.195) evicted. New cstate: current_term: 2 leader_uuid: "3ce79ee2bb1348d6a36d8ce39df7d3aa" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "3ce79ee2bb1348d6a36d8ce39df7d3aa" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 44481 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d9fd45507a8e435abc710b00a27538c6" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 44183 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b5c46129f1994ab28f3710e8e392ceab" member_type: VOTER last_known_addr { host: "127.0.57.196" port: 34535 } attrs { promote: false } health_report { overall_health: HEALTHY } } } I0507 06:32:20.021040 9388 catalog_manager.cc:3168] Scheduling retry of 3a7d8f424df742599d3f6d48d9b1a37d Delete Tablet RPC for TS=2781869a927549908dde07bc5019d0e4 with a delay of 54 ms (attempt = 1) W0507 06:32:20.021158 9388 catalog_manager.cc:4147] Failed to send 3a7d8f424df742599d3f6d48d9b1a37d Delete Tablet RPC for TS=2781869a927549908dde07bc5019d0e4: Not found: failed to reset TS proxy: Could not find TS for UUID 2781869a927549908dde07bc5019d0e4 I0507 06:32:20.048279 9372 catalog_manager.cc:3564] ChangeConfig:REMOVE_PEER RPC for tablet 3a7d8f424df742599d3f6d48d9b1a37d with cas_config_opid_index 10: aborting the task: latest config opid_index 11; task opid_index 10 I0507 06:32:20.075390 9375 catalog_manager.cc:3168] Scheduling retry of 3a7d8f424df742599d3f6d48d9b1a37d Delete Tablet RPC for TS=2781869a927549908dde07bc5019d0e4 with a delay of 58 ms (attempt = 2) W0507 06:32:20.075498 9375 catalog_manager.cc:3187] Async tablet task 3a7d8f424df742599d3f6d48d9b1a37d Delete Tablet RPC for TS=2781869a927549908dde07bc5019d0e4 failed: Not found: failed to reset TS proxy: Could not find TS for UUID 2781869a927549908dde07bc5019d0e4 I0507 06:32:20.134399 9375 catalog_manager.cc:3168] Scheduling retry of 3a7d8f424df742599d3f6d48d9b1a37d Delete Tablet RPC for TS=2781869a927549908dde07bc5019d0e4 with a delay of 97 ms (attempt = 3) W0507 06:32:20.134505 9375 catalog_manager.cc:3187] Async tablet task 3a7d8f424df742599d3f6d48d9b1a37d Delete Tablet RPC for TS=2781869a927549908dde07bc5019d0e4 failed: Not found: failed to reset TS proxy: Could not find TS for UUID 2781869a927549908dde07bc5019d0e4 I0507 06:32:20.232038 9375 catalog_manager.cc:3168] Scheduling retry of 3a7d8f424df742599d3f6d48d9b1a37d Delete Tablet RPC for TS=2781869a927549908dde07bc5019d0e4 with a delay of 147 ms (attempt = 4) W0507 06:32:20.232159 9375 catalog_manager.cc:3187] Async tablet task 3a7d8f424df742599d3f6d48d9b1a37d Delete Tablet RPC for TS=2781869a927549908dde07bc5019d0e4 failed: Not found: failed to reset TS proxy: Could not find TS for UUID 2781869a927549908dde07bc5019d0e4 I0507 06:32:20.259017 9355 heartbeater.cc:345] Connected to a master server at 127.0.57.254:34231 I0507 06:32:20.259208 9355 heartbeater.cc:426] Registering TS with master... I0507 06:32:20.259801 9355 heartbeater.cc:475] Master 127.0.57.254:34231 requested a full tablet report, sending... I0507 06:32:20.261019 9390 ts_manager.cc:159] Registered new tserver with Master: 44a1757d4e604357a1991e808f009e56 (127.0.57.199:44671) I0507 06:32:20.262061 9390 master_service.cc:273] Signed X509 certificate for tserver {username='slave'} at 127.0.57.199:33387 I0507 06:32:20.263758 231 external_mini_cluster.cc:623] 6 TS(s) registered with all masters I0507 06:32:20.266971 231 ts_itest-base.cc:232] Waiting for 1 tablets on tserver 3ce79ee2bb1348d6a36d8ce39df7d3aa to finish bootstrapping I0507 06:32:20.274264 231 ts_itest-base.cc:232] Waiting for 1 tablets on tserver d9fd45507a8e435abc710b00a27538c6 to finish bootstrapping I0507 06:32:20.280856 231 ts_itest-base.cc:232] Waiting for 0 tablets on tserver 2781869a927549908dde07bc5019d0e4 to finish bootstrapping I0507 06:32:20.281888 231 ts_itest-base.cc:232] Waiting for 1 tablets on tserver b5c46129f1994ab28f3710e8e392ceab to finish bootstrapping I0507 06:32:20.295589 231 ts_itest-base.cc:232] Waiting for 0 tablets on tserver bb5a49b22d4048608d9e7b762d749fb2 to finish bootstrapping I0507 06:32:20.305944 231 ts_itest-base.cc:232] Waiting for 0 tablets on tserver 3e033fe6ec6f4fe595bdcd8a95a9f4e8 to finish bootstrapping I0507 06:32:20.319785 231 ts_itest-base.cc:232] Waiting for 0 tablets on tserver 44a1757d4e604357a1991e808f009e56 to finish bootstrapping I0507 06:32:20.346386 231 external_mini_cluster.cc:1172] Attempting to check leaks for /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver pid 8615 I0507 06:32:20.347254 8733 generic_service.cc:171] Checking for leaks (request via RPC) I0507 06:32:20.379464 9375 catalog_manager.cc:3168] Scheduling retry of 3a7d8f424df742599d3f6d48d9b1a37d Delete Tablet RPC for TS=2781869a927549908dde07bc5019d0e4 with a delay of 289 ms (attempt = 5) W0507 06:32:20.379613 9375 catalog_manager.cc:3187] Async tablet task 3a7d8f424df742599d3f6d48d9b1a37d Delete Tablet RPC for TS=2781869a927549908dde07bc5019d0e4 failed: Not found: failed to reset TS proxy: Could not find TS for UUID 2781869a927549908dde07bc5019d0e4 I0507 06:32:20.398914 231 external_mini_cluster.cc:1131] Killing /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver with pid 8615 I0507 06:32:20.412253 231 external_mini_cluster.cc:1172] Attempting to check leaks for /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver pid 8740 I0507 06:32:20.413338 8858 generic_service.cc:171] Checking for leaks (request via RPC) I0507 06:32:20.465009 231 external_mini_cluster.cc:1131] Killing /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver with pid 8740 I0507 06:32:20.475966 231 external_mini_cluster.cc:1172] Attempting to check leaks for /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver pid 8865 I0507 06:32:20.477236 8983 generic_service.cc:171] Checking for leaks (request via RPC) I0507 06:32:20.529021 231 external_mini_cluster.cc:1131] Killing /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver with pid 8865 I0507 06:32:20.539583 231 external_mini_cluster.cc:1172] Attempting to check leaks for /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver pid 8989 I0507 06:32:20.540825 9106 generic_service.cc:171] Checking for leaks (request via RPC) I0507 06:32:20.584180 231 external_mini_cluster.cc:1131] Killing /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver with pid 8989 I0507 06:32:20.595084 231 external_mini_cluster.cc:1172] Attempting to check leaks for /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver pid 9112 I0507 06:32:20.596330 9229 generic_service.cc:171] Checking for leaks (request via RPC) I0507 06:32:20.638984 231 external_mini_cluster.cc:1131] Killing /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver with pid 9112 I0507 06:32:20.648835 231 external_mini_cluster.cc:1172] Attempting to check leaks for /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver pid 9235 I0507 06:32:20.650066 9352 generic_service.cc:171] Checking for leaks (request via RPC) I0507 06:32:20.669232 9375 catalog_manager.cc:3168] Scheduling retry of 3a7d8f424df742599d3f6d48d9b1a37d Delete Tablet RPC for TS=2781869a927549908dde07bc5019d0e4 with a delay of 555 ms (attempt = 6) W0507 06:32:20.669378 9375 catalog_manager.cc:3187] Async tablet task 3a7d8f424df742599d3f6d48d9b1a37d Delete Tablet RPC for TS=2781869a927549908dde07bc5019d0e4 failed: Not found: failed to reset TS proxy: Could not find TS for UUID 2781869a927549908dde07bc5019d0e4 I0507 06:32:20.695389 231 external_mini_cluster.cc:1131] Killing /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver with pid 9235 I0507 06:32:20.704874 231 external_mini_cluster.cc:1172] Attempting to check leaks for /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-master pid 9358 I0507 06:32:20.706074 9421 generic_service.cc:171] Checking for leaks (request via RPC) I0507 06:32:20.792829 231 external_mini_cluster.cc:1131] Killing /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-master with pid 9358 [ OK ] RaftConsensusNonVoterITest.RestartClusterWithNonVoter (19702 ms) [ RUN ] RaftConsensusNonVoterITest.NonVoterReplicasInConsensusQueue I0507 06:32:20.817452 231 test_util.cc:212] Using random seed: 1628175898 I0507 06:32:20.819054 231 ts_itest-base.cc:112] Starting cluster with: I0507 06:32:20.819095 231 ts_itest-base.cc:113] -------------- I0507 06:32:20.819125 231 ts_itest-base.cc:114] 4 tablet servers I0507 06:32:20.819175 231 ts_itest-base.cc:115] 3 replicas per TS I0507 06:32:20.819203 231 ts_itest-base.cc:116] -------------- I0507 06:32:20.820413 231 external_mini_cluster.cc:933] Running /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-master /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-master --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.NonVoterReplicasInConsensusQueue.1557210500371863-231/raft_consensus-itest-cluster/master-0/wal --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.NonVoterReplicasInConsensusQueue.1557210500371863-231/raft_consensus-itest-cluster/master-0/data --block_manager=log --webserver_interface=localhost --ipki_ca_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.0.57.254:32871 --webserver_interface=127.0.57.254 --webserver_port=0 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.NonVoterReplicasInConsensusQueue.1557210500371863-231/raft_consensus-itest-cluster/master-0/logs --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.NonVoterReplicasInConsensusQueue.1557210500371863-231/raft_consensus-itest-cluster/master-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --catalog_manager_evict_excess_replicas=false --rpc_reuseport=true with env {} WARNING: Logging before InitGoogleLogging() is written to STDERR W0507 06:32:21.034487 9473 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0507 06:32:21.034723 9473 flags.cc:404] Enabled unsafe flag: --never_fsync=true W0507 06:32:21.037204 9473 flags.cc:404] Enabled experimental flag: --ipki_ca_key_size=1024 W0507 06:32:21.037250 9473 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W0507 06:32:21.037282 9473 flags.cc:404] Enabled experimental flag: --tsk_num_rsa_bits=512 W0507 06:32:21.037307 9473 flags.cc:404] Enabled experimental flag: --rpc_reuseport=true I0507 06:32:21.041946 9473 master_main.cc:73] Master server non-default flags: --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.NonVoterReplicasInConsensusQueue.1557210500371863-231/raft_consensus-itest-cluster/master-0/data --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.NonVoterReplicasInConsensusQueue.1557210500371863-231/raft_consensus-itest-cluster/master-0/wal --catalog_manager_evict_excess_replicas=false --ipki_ca_key_size=1024 --ipki_server_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.0.57.254:32871 --rpc_reuseport=true --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.NonVoterReplicasInConsensusQueue.1557210500371863-231/raft_consensus-itest-cluster/master-0/data/info.pb --webserver_interface=127.0.57.254 --webserver_port=0 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.NonVoterReplicasInConsensusQueue.1557210500371863-231/raft_consensus-itest-cluster/master-0/logs --logbuflevel=-1 --logtostderr=true Master server version: kudu 1.10.0-cdh6.x-SNAPSHOT revision 7d0ee26697926e87af9ef395008799577f6ab43f build type FASTDEBUG built by jenkins at 06 May 2019 22:44:21 PST on quasar-knmpfx-1.vpc.cloudera.com build id 322017 ASAN enabled I0507 06:32:21.049293 9473 system_ntp.cc:202] NTP initialized. Skew: 500ppm Current error: 145016us I0507 06:32:21.050585 9473 fs_manager.cc:263] Metadata directory not provided I0507 06:32:21.050665 9473 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0507 06:32:21.050935 9473 server_base.cc:433] Could not load existing FS layout: Not found: could not find a healthy instance file I0507 06:32:21.051013 9473 server_base.cc:434] Attempting to create new FS layout instead I0507 06:32:21.053580 9473 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.NonVoterReplicasInConsensusQueue.1557210500371863-231/raft_consensus-itest-cluster/master-0/data/instance: uuid: "869bbb5f4ade459caa8880b2092b95a0" format_stamp: "Formatted at 2019-05-07 06:32:21 on dist-test-slave-dist-test-slave-rqp6" I0507 06:32:21.054113 9473 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.NonVoterReplicasInConsensusQueue.1557210500371863-231/raft_consensus-itest-cluster/master-0/wal/instance: uuid: "869bbb5f4ade459caa8880b2092b95a0" format_stamp: "Formatted at 2019-05-07 06:32:21 on dist-test-slave-dist-test-slave-rqp6" I0507 06:32:21.061246 9473 fs_manager.cc:503] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s I0507 06:32:21.062332 9473 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0507 06:32:21.062878 9473 file_cache.cc:466] Constructed file cache lbm with capacity 419430 I0507 06:32:21.066452 9473 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s I0507 06:32:21.066524 9473 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.NonVoterReplicasInConsensusQueue.1557210500371863-231/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.NonVoterReplicasInConsensusQueue.1557210500371863-231/raft_consensus-itest-cluster/master-0/wal uuid: "869bbb5f4ade459caa8880b2092b95a0" format_stamp: "Formatted at 2019-05-07 06:32:21 on dist-test-slave-dist-test-slave-rqp6" I0507 06:32:21.066676 9473 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.NonVoterReplicasInConsensusQueue.1557210500371863-231/raft_consensus-itest-cluster/master-0/wal metadata directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.NonVoterReplicasInConsensusQueue.1557210500371863-231/raft_consensus-itest-cluster/master-0/wal 1 data directories: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.NonVoterReplicasInConsensusQueue.1557210500371863-231/raft_consensus-itest-cluster/master-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0507 06:32:21.109338 9473 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0507 06:32:21.125461 9473 rpc_server.cc:205] RPC server started. Bound to: 127.0.57.254:32871 I0507 06:32:21.126628 9473 webserver.cc:308] Webserver started at http://127.0.57.254:36647/ using document root and password file I0507 06:32:21.127365 9473 server_base.cc:608] Dumped server information to /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.NonVoterReplicasInConsensusQueue.1557210500371863-231/raft_consensus-itest-cluster/master-0/data/info.pb I0507 06:32:21.130695 231 external_mini_cluster.cc:995] Started /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-master as pid 9473 I0507 06:32:21.131347 9532 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0507 06:32:21.131466 231 external_mini_cluster.cc:933] Running /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.NonVoterReplicasInConsensusQueue.1557210500371863-231/raft_consensus-itest-cluster/ts-0/wal --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.NonVoterReplicasInConsensusQueue.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data --block_manager=log --rpc_bind_addresses=127.0.57.193:0 --local_ip_for_outbound_sockets=127.0.57.193 --webserver_interface=127.0.57.193 --webserver_port=0 --tserver_master_addrs=127.0.57.254:32871 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.NonVoterReplicasInConsensusQueue.1557210500371863-231/raft_consensus-itest-cluster/ts-0/logs --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.NonVoterReplicasInConsensusQueue.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_heartbeat_interval_ms=50 with env {} I0507 06:32:21.143371 9532 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 869bbb5f4ade459caa8880b2092b95a0: Bootstrap starting. I0507 06:32:21.146497 9532 tablet_bootstrap.cc:590] T 00000000000000000000000000000000 P 869bbb5f4ade459caa8880b2092b95a0: No blocks or log segments found. Creating new log. I0507 06:32:21.147526 9532 log.cc:525] T 00000000000000000000000000000000 P 869bbb5f4ade459caa8880b2092b95a0: Log is configured to *not* fsync() on all Append() calls I0507 06:32:21.149400 9532 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 869bbb5f4ade459caa8880b2092b95a0: No bootstrap required, opened a new log I0507 06:32:21.156302 9532 raft_consensus.cc:340] T 00000000000000000000000000000000 P 869bbb5f4ade459caa8880b2092b95a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "869bbb5f4ade459caa8880b2092b95a0" member_type: VOTER } I0507 06:32:21.156607 9532 raft_consensus.cc:366] T 00000000000000000000000000000000 P 869bbb5f4ade459caa8880b2092b95a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0507 06:32:21.156677 9532 raft_consensus.cc:705] T 00000000000000000000000000000000 P 869bbb5f4ade459caa8880b2092b95a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 869bbb5f4ade459caa8880b2092b95a0, State: Initialized, Role: FOLLOWER I0507 06:32:21.157282 9532 consensus_queue.cc:229] T 00000000000000000000000000000000 P 869bbb5f4ade459caa8880b2092b95a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "869bbb5f4ade459caa8880b2092b95a0" member_type: VOTER } I0507 06:32:21.157471 9532 raft_consensus.cc:380] T 00000000000000000000000000000000 P 869bbb5f4ade459caa8880b2092b95a0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0507 06:32:21.157562 9532 raft_consensus.cc:472] T 00000000000000000000000000000000 P 869bbb5f4ade459caa8880b2092b95a0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0507 06:32:21.157666 9532 raft_consensus.cc:2886] T 00000000000000000000000000000000 P 869bbb5f4ade459caa8880b2092b95a0 [term 0 FOLLOWER]: Advancing to term 1 I0507 06:32:21.159405 9532 raft_consensus.cc:494] T 00000000000000000000000000000000 P 869bbb5f4ade459caa8880b2092b95a0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "869bbb5f4ade459caa8880b2092b95a0" member_type: VOTER } I0507 06:32:21.159886 9532 leader_election.cc:310] T 00000000000000000000000000000000 P 869bbb5f4ade459caa8880b2092b95a0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 869bbb5f4ade459caa8880b2092b95a0; no voters: I0507 06:32:21.160610 9532 leader_election.cc:296] T 00000000000000000000000000000000 P 869bbb5f4ade459caa8880b2092b95a0 [CANDIDATE]: Term 1 election: Requested vote from peers I0507 06:32:21.160782 9535 raft_consensus.cc:2640] T 00000000000000000000000000000000 P 869bbb5f4ade459caa8880b2092b95a0 [term 1 FOLLOWER]: Leader election won for term 1 I0507 06:32:21.165596 9535 raft_consensus.cc:667] T 00000000000000000000000000000000 P 869bbb5f4ade459caa8880b2092b95a0 [term 1 LEADER]: Becoming Leader. State: Replica: 869bbb5f4ade459caa8880b2092b95a0, State: Running, Role: LEADER I0507 06:32:21.166038 9535 consensus_queue.cc:206] T 00000000000000000000000000000000 P 869bbb5f4ade459caa8880b2092b95a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "869bbb5f4ade459caa8880b2092b95a0" member_type: VOTER } I0507 06:32:21.169709 9537 sys_catalog.cc:337] T 00000000000000000000000000000000 P 869bbb5f4ade459caa8880b2092b95a0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 869bbb5f4ade459caa8880b2092b95a0. Latest consensus state: current_term: 1 leader_uuid: "869bbb5f4ade459caa8880b2092b95a0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "869bbb5f4ade459caa8880b2092b95a0" member_type: VOTER } } I0507 06:32:21.170075 9537 sys_catalog.cc:340] T 00000000000000000000000000000000 P 869bbb5f4ade459caa8880b2092b95a0 [sys.catalog]: This master's current role is: LEADER I0507 06:32:21.173802 9536 sys_catalog.cc:337] T 00000000000000000000000000000000 P 869bbb5f4ade459caa8880b2092b95a0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "869bbb5f4ade459caa8880b2092b95a0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "869bbb5f4ade459caa8880b2092b95a0" member_type: VOTER } } I0507 06:32:21.174037 9536 sys_catalog.cc:340] T 00000000000000000000000000000000 P 869bbb5f4ade459caa8880b2092b95a0 [sys.catalog]: This master's current role is: LEADER I0507 06:32:21.173964 9532 sys_catalog.cc:337] T 00000000000000000000000000000000 P 869bbb5f4ade459caa8880b2092b95a0 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 1 leader_uuid: "869bbb5f4ade459caa8880b2092b95a0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "869bbb5f4ade459caa8880b2092b95a0" member_type: VOTER } } I0507 06:32:21.174132 9532 sys_catalog.cc:340] T 00000000000000000000000000000000 P 869bbb5f4ade459caa8880b2092b95a0 [sys.catalog]: This master's current role is: LEADER I0507 06:32:21.175525 9532 sys_catalog.cc:424] T 00000000000000000000000000000000 P 869bbb5f4ade459caa8880b2092b95a0 [sys.catalog]: configured and running, proceeding with master startup. I0507 06:32:21.175724 9541 catalog_manager.cc:1055] Loading table and tablet metadata into memory... I0507 06:32:21.179940 9541 catalog_manager.cc:1066] Initializing Kudu internal certificate authority... I0507 06:32:21.242975 9541 catalog_manager.cc:950] Generated new certificate authority record I0507 06:32:21.244354 9541 catalog_manager.cc:1075] Loading token signing keys... I0507 06:32:21.261162 9541 catalog_manager.cc:4292] T 00000000000000000000000000000000 P 869bbb5f4ade459caa8880b2092b95a0: Generated new TSK 0 WARNING: Logging before InitGoogleLogging() is written to STDERR W0507 06:32:21.295680 9533 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0507 06:32:21.295888 9533 flags.cc:404] Enabled unsafe flag: --never_fsync=true W0507 06:32:21.297981 9533 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W0507 06:32:21.298101 9533 flags.cc:404] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.57.193 I0507 06:32:21.302785 9533 tablet_server_main.cc:78] Tablet server non-default flags: --raft_heartbeat_interval_ms=50 --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.NonVoterReplicasInConsensusQueue.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.NonVoterReplicasInConsensusQueue.1557210500371863-231/raft_consensus-itest-cluster/ts-0/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.0.57.193:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.NonVoterReplicasInConsensusQueue.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data/info.pb --webserver_interface=127.0.57.193 --webserver_port=0 --tserver_master_addrs=127.0.57.254:32871 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.0.57.193 --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.NonVoterReplicasInConsensusQueue.1557210500371863-231/raft_consensus-itest-cluster/ts-0/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.10.0-cdh6.x-SNAPSHOT revision 7d0ee26697926e87af9ef395008799577f6ab43f build type FASTDEBUG built by jenkins at 06 May 2019 22:44:21 PST on quasar-knmpfx-1.vpc.cloudera.com build id 322017 ASAN enabled I0507 06:32:21.309885 9533 system_ntp.cc:202] NTP initialized. Skew: 500ppm Current error: 145016us I0507 06:32:21.310536 9533 fs_manager.cc:263] Metadata directory not provided I0507 06:32:21.310612 9533 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0507 06:32:21.310871 9533 server_base.cc:433] Could not load existing FS layout: Not found: could not find a healthy instance file I0507 06:32:21.310947 9533 server_base.cc:434] Attempting to create new FS layout instead I0507 06:32:21.313390 9533 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.NonVoterReplicasInConsensusQueue.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data/instance: uuid: "b46048beca56499c8ee771446de3cbd4" format_stamp: "Formatted at 2019-05-07 06:32:21 on dist-test-slave-dist-test-slave-rqp6" I0507 06:32:21.313916 9533 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.NonVoterReplicasInConsensusQueue.1557210500371863-231/raft_consensus-itest-cluster/ts-0/wal/instance: uuid: "b46048beca56499c8ee771446de3cbd4" format_stamp: "Formatted at 2019-05-07 06:32:21 on dist-test-slave-dist-test-slave-rqp6" I0507 06:32:21.320040 9533 fs_manager.cc:503] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s I0507 06:32:21.321060 9533 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0507 06:32:21.321584 9533 file_cache.cc:466] Constructed file cache lbm with capacity 419430 I0507 06:32:21.325073 9533 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s I0507 06:32:21.325145 9533 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.NonVoterReplicasInConsensusQueue.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.NonVoterReplicasInConsensusQueue.1557210500371863-231/raft_consensus-itest-cluster/ts-0/wal uuid: "b46048beca56499c8ee771446de3cbd4" format_stamp: "Formatted at 2019-05-07 06:32:21 on dist-test-slave-dist-test-slave-rqp6" I0507 06:32:21.325330 9533 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.NonVoterReplicasInConsensusQueue.1557210500371863-231/raft_consensus-itest-cluster/ts-0/wal metadata directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.NonVoterReplicasInConsensusQueue.1557210500371863-231/raft_consensus-itest-cluster/ts-0/wal 1 data directories: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.NonVoterReplicasInConsensusQueue.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0507 06:32:21.355743 9533 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0507 06:32:21.357616 9533 ts_tablet_manager.cc:356] Loaded tablet metadata (0 total tablets, 0 live tablets) I0507 06:32:21.357683 9533 ts_tablet_manager.cc:376] Registered 0 tablets I0507 06:32:21.441715 9533 rpc_server.cc:205] RPC server started. Bound to: 127.0.57.193:46277 I0507 06:32:21.443096 9533 webserver.cc:308] Webserver started at http://127.0.57.193:44431/ using document root and password file I0507 06:32:21.443886 9533 server_base.cc:608] Dumped server information to /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.NonVoterReplicasInConsensusQueue.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data/info.pb I0507 06:32:21.453562 231 external_mini_cluster.cc:995] Started /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver as pid 9533 I0507 06:32:21.454228 231 external_mini_cluster.cc:933] Running /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.NonVoterReplicasInConsensusQueue.1557210500371863-231/raft_consensus-itest-cluster/ts-1/wal --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.NonVoterReplicasInConsensusQueue.1557210500371863-231/raft_consensus-itest-cluster/ts-1/data --block_manager=log --rpc_bind_addresses=127.0.57.194:0 --local_ip_for_outbound_sockets=127.0.57.194 --webserver_interface=127.0.57.194 --webserver_port=0 --tserver_master_addrs=127.0.57.254:32871 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.NonVoterReplicasInConsensusQueue.1557210500371863-231/raft_consensus-itest-cluster/ts-1/logs --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.NonVoterReplicasInConsensusQueue.1557210500371863-231/raft_consensus-itest-cluster/ts-1/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_heartbeat_interval_ms=50 with env {} I0507 06:32:21.471127 9663 heartbeater.cc:345] Connected to a master server at 127.0.57.254:32871 I0507 06:32:21.471350 9663 heartbeater.cc:426] Registering TS with master... I0507 06:32:21.471928 9663 heartbeater.cc:475] Master 127.0.57.254:32871 requested a full tablet report, sending... I0507 06:32:21.473752 9499 ts_manager.cc:159] Registered new tserver with Master: b46048beca56499c8ee771446de3cbd4 (127.0.57.193:46277) I0507 06:32:21.475435 9499 master_service.cc:273] Signed X509 certificate for tserver {username='slave'} at 127.0.57.193:47581 WARNING: Logging before InitGoogleLogging() is written to STDERR W0507 06:32:21.612200 9667 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0507 06:32:21.612409 9667 flags.cc:404] Enabled unsafe flag: --never_fsync=true W0507 06:32:21.614454 9667 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W0507 06:32:21.614549 9667 flags.cc:404] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.57.194 I0507 06:32:21.618356 9667 tablet_server_main.cc:78] Tablet server non-default flags: --raft_heartbeat_interval_ms=50 --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.NonVoterReplicasInConsensusQueue.1557210500371863-231/raft_consensus-itest-cluster/ts-1/data --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.NonVoterReplicasInConsensusQueue.1557210500371863-231/raft_consensus-itest-cluster/ts-1/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.0.57.194:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.NonVoterReplicasInConsensusQueue.1557210500371863-231/raft_consensus-itest-cluster/ts-1/data/info.pb --webserver_interface=127.0.57.194 --webserver_port=0 --tserver_master_addrs=127.0.57.254:32871 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.0.57.194 --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.NonVoterReplicasInConsensusQueue.1557210500371863-231/raft_consensus-itest-cluster/ts-1/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.10.0-cdh6.x-SNAPSHOT revision 7d0ee26697926e87af9ef395008799577f6ab43f build type FASTDEBUG built by jenkins at 06 May 2019 22:44:21 PST on quasar-knmpfx-1.vpc.cloudera.com build id 322017 ASAN enabled I0507 06:32:21.626807 9667 system_ntp.cc:202] NTP initialized. Skew: 500ppm Current error: 145016us I0507 06:32:21.627530 9667 fs_manager.cc:263] Metadata directory not provided I0507 06:32:21.627617 9667 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0507 06:32:21.627882 9667 server_base.cc:433] Could not load existing FS layout: Not found: could not find a healthy instance file I0507 06:32:21.627960 9667 server_base.cc:434] Attempting to create new FS layout instead I0507 06:32:21.630668 9667 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.NonVoterReplicasInConsensusQueue.1557210500371863-231/raft_consensus-itest-cluster/ts-1/data/instance: uuid: "0198be74df5a45dea84591d6b6c20c66" format_stamp: "Formatted at 2019-05-07 06:32:21 on dist-test-slave-dist-test-slave-rqp6" I0507 06:32:21.631265 9667 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.NonVoterReplicasInConsensusQueue.1557210500371863-231/raft_consensus-itest-cluster/ts-1/wal/instance: uuid: "0198be74df5a45dea84591d6b6c20c66" format_stamp: "Formatted at 2019-05-07 06:32:21 on dist-test-slave-dist-test-slave-rqp6" I0507 06:32:21.638057 9667 fs_manager.cc:503] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s I0507 06:32:21.639127 9667 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0507 06:32:21.639674 9667 file_cache.cc:466] Constructed file cache lbm with capacity 419430 I0507 06:32:21.643123 9667 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s I0507 06:32:21.643200 9667 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.NonVoterReplicasInConsensusQueue.1557210500371863-231/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.NonVoterReplicasInConsensusQueue.1557210500371863-231/raft_consensus-itest-cluster/ts-1/wal uuid: "0198be74df5a45dea84591d6b6c20c66" format_stamp: "Formatted at 2019-05-07 06:32:21 on dist-test-slave-dist-test-slave-rqp6" I0507 06:32:21.643368 9667 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.NonVoterReplicasInConsensusQueue.1557210500371863-231/raft_consensus-itest-cluster/ts-1/wal metadata directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.NonVoterReplicasInConsensusQueue.1557210500371863-231/raft_consensus-itest-cluster/ts-1/wal 1 data directories: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.NonVoterReplicasInConsensusQueue.1557210500371863-231/raft_consensus-itest-cluster/ts-1/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0507 06:32:21.686656 9667 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0507 06:32:21.688431 9667 ts_tablet_manager.cc:356] Loaded tablet metadata (0 total tablets, 0 live tablets) I0507 06:32:21.688495 9667 ts_tablet_manager.cc:376] Registered 0 tablets I0507 06:32:21.715680 9667 rpc_server.cc:205] RPC server started. Bound to: 127.0.57.194:40207 I0507 06:32:21.716887 9667 webserver.cc:308] Webserver started at http://127.0.57.194:40567/ using document root and password file I0507 06:32:21.717583 9667 server_base.cc:608] Dumped server information to /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.NonVoterReplicasInConsensusQueue.1557210500371863-231/raft_consensus-itest-cluster/ts-1/data/info.pb I0507 06:32:21.725677 231 external_mini_cluster.cc:995] Started /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver as pid 9667 I0507 06:32:21.726418 231 external_mini_cluster.cc:933] Running /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.NonVoterReplicasInConsensusQueue.1557210500371863-231/raft_consensus-itest-cluster/ts-2/wal --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.NonVoterReplicasInConsensusQueue.1557210500371863-231/raft_consensus-itest-cluster/ts-2/data --block_manager=log --rpc_bind_addresses=127.0.57.195:0 --local_ip_for_outbound_sockets=127.0.57.195 --webserver_interface=127.0.57.195 --webserver_port=0 --tserver_master_addrs=127.0.57.254:32871 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.NonVoterReplicasInConsensusQueue.1557210500371863-231/raft_consensus-itest-cluster/ts-2/logs --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.NonVoterReplicasInConsensusQueue.1557210500371863-231/raft_consensus-itest-cluster/ts-2/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_heartbeat_interval_ms=50 with env {} I0507 06:32:21.731271 9786 heartbeater.cc:345] Connected to a master server at 127.0.57.254:32871 I0507 06:32:21.731496 9786 heartbeater.cc:426] Registering TS with master... I0507 06:32:21.732156 9786 heartbeater.cc:475] Master 127.0.57.254:32871 requested a full tablet report, sending... I0507 06:32:21.733495 9499 ts_manager.cc:159] Registered new tserver with Master: 0198be74df5a45dea84591d6b6c20c66 (127.0.57.194:40207) I0507 06:32:21.734493 9499 master_service.cc:273] Signed X509 certificate for tserver {username='slave'} at 127.0.57.194:58345 WARNING: Logging before InitGoogleLogging() is written to STDERR W0507 06:32:21.888195 9789 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0507 06:32:21.888411 9789 flags.cc:404] Enabled unsafe flag: --never_fsync=true W0507 06:32:21.890463 9789 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W0507 06:32:21.890565 9789 flags.cc:404] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.57.195 I0507 06:32:21.894394 9789 tablet_server_main.cc:78] Tablet server non-default flags: --raft_heartbeat_interval_ms=50 --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.NonVoterReplicasInConsensusQueue.1557210500371863-231/raft_consensus-itest-cluster/ts-2/data --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.NonVoterReplicasInConsensusQueue.1557210500371863-231/raft_consensus-itest-cluster/ts-2/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.0.57.195:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.NonVoterReplicasInConsensusQueue.1557210500371863-231/raft_consensus-itest-cluster/ts-2/data/info.pb --webserver_interface=127.0.57.195 --webserver_port=0 --tserver_master_addrs=127.0.57.254:32871 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.0.57.195 --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.NonVoterReplicasInConsensusQueue.1557210500371863-231/raft_consensus-itest-cluster/ts-2/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.10.0-cdh6.x-SNAPSHOT revision 7d0ee26697926e87af9ef395008799577f6ab43f build type FASTDEBUG built by jenkins at 06 May 2019 22:44:21 PST on quasar-knmpfx-1.vpc.cloudera.com build id 322017 ASAN enabled I0507 06:32:21.901614 9789 system_ntp.cc:202] NTP initialized. Skew: 500ppm Current error: 145016us I0507 06:32:21.902246 9789 fs_manager.cc:263] Metadata directory not provided I0507 06:32:21.902324 9789 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0507 06:32:21.902559 9789 server_base.cc:433] Could not load existing FS layout: Not found: could not find a healthy instance file I0507 06:32:21.902637 9789 server_base.cc:434] Attempting to create new FS layout instead I0507 06:32:21.904968 9789 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.NonVoterReplicasInConsensusQueue.1557210500371863-231/raft_consensus-itest-cluster/ts-2/data/instance: uuid: "a6726eec155e4dc48c6ae00643563ab4" format_stamp: "Formatted at 2019-05-07 06:32:21 on dist-test-slave-dist-test-slave-rqp6" I0507 06:32:21.905503 9789 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.NonVoterReplicasInConsensusQueue.1557210500371863-231/raft_consensus-itest-cluster/ts-2/wal/instance: uuid: "a6726eec155e4dc48c6ae00643563ab4" format_stamp: "Formatted at 2019-05-07 06:32:21 on dist-test-slave-dist-test-slave-rqp6" I0507 06:32:21.911588 9789 fs_manager.cc:503] Time spent creating directory manager: real 0.006s user 0.001s sys 0.004s I0507 06:32:21.912571 9789 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0507 06:32:21.913113 9789 file_cache.cc:466] Constructed file cache lbm with capacity 419430 I0507 06:32:21.916136 9789 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s I0507 06:32:21.916208 9789 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.NonVoterReplicasInConsensusQueue.1557210500371863-231/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.NonVoterReplicasInConsensusQueue.1557210500371863-231/raft_consensus-itest-cluster/ts-2/wal uuid: "a6726eec155e4dc48c6ae00643563ab4" format_stamp: "Formatted at 2019-05-07 06:32:21 on dist-test-slave-dist-test-slave-rqp6" I0507 06:32:21.916365 9789 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.NonVoterReplicasInConsensusQueue.1557210500371863-231/raft_consensus-itest-cluster/ts-2/wal metadata directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.NonVoterReplicasInConsensusQueue.1557210500371863-231/raft_consensus-itest-cluster/ts-2/wal 1 data directories: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.NonVoterReplicasInConsensusQueue.1557210500371863-231/raft_consensus-itest-cluster/ts-2/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0507 06:32:21.953279 9789 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0507 06:32:21.954924 9789 ts_tablet_manager.cc:356] Loaded tablet metadata (0 total tablets, 0 live tablets) I0507 06:32:21.955022 9789 ts_tablet_manager.cc:376] Registered 0 tablets I0507 06:32:21.982179 9789 rpc_server.cc:205] RPC server started. Bound to: 127.0.57.195:38581 I0507 06:32:21.983436 9789 webserver.cc:308] Webserver started at http://127.0.57.195:44847/ using document root and password file I0507 06:32:21.984185 9789 server_base.cc:608] Dumped server information to /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.NonVoterReplicasInConsensusQueue.1557210500371863-231/raft_consensus-itest-cluster/ts-2/data/info.pb I0507 06:32:21.988219 231 external_mini_cluster.cc:995] Started /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver as pid 9789 I0507 06:32:21.988960 231 external_mini_cluster.cc:933] Running /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.NonVoterReplicasInConsensusQueue.1557210500371863-231/raft_consensus-itest-cluster/ts-3/wal --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.NonVoterReplicasInConsensusQueue.1557210500371863-231/raft_consensus-itest-cluster/ts-3/data --block_manager=log --rpc_bind_addresses=127.0.57.196:0 --local_ip_for_outbound_sockets=127.0.57.196 --webserver_interface=127.0.57.196 --webserver_port=0 --tserver_master_addrs=127.0.57.254:32871 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.NonVoterReplicasInConsensusQueue.1557210500371863-231/raft_consensus-itest-cluster/ts-3/logs --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.NonVoterReplicasInConsensusQueue.1557210500371863-231/raft_consensus-itest-cluster/ts-3/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_heartbeat_interval_ms=50 with env {} I0507 06:32:21.996004 9908 heartbeater.cc:345] Connected to a master server at 127.0.57.254:32871 I0507 06:32:21.996209 9908 heartbeater.cc:426] Registering TS with master... I0507 06:32:21.996700 9908 heartbeater.cc:475] Master 127.0.57.254:32871 requested a full tablet report, sending... I0507 06:32:21.997880 9499 ts_manager.cc:159] Registered new tserver with Master: a6726eec155e4dc48c6ae00643563ab4 (127.0.57.195:38581) I0507 06:32:21.998987 9499 master_service.cc:273] Signed X509 certificate for tserver {username='slave'} at 127.0.57.195:52529 WARNING: Logging before InitGoogleLogging() is written to STDERR W0507 06:32:22.144891 9911 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0507 06:32:22.145104 9911 flags.cc:404] Enabled unsafe flag: --never_fsync=true W0507 06:32:22.147107 9911 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W0507 06:32:22.147203 9911 flags.cc:404] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.57.196 I0507 06:32:22.150990 9911 tablet_server_main.cc:78] Tablet server non-default flags: --raft_heartbeat_interval_ms=50 --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.NonVoterReplicasInConsensusQueue.1557210500371863-231/raft_consensus-itest-cluster/ts-3/data --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.NonVoterReplicasInConsensusQueue.1557210500371863-231/raft_consensus-itest-cluster/ts-3/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.0.57.196:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.NonVoterReplicasInConsensusQueue.1557210500371863-231/raft_consensus-itest-cluster/ts-3/data/info.pb --webserver_interface=127.0.57.196 --webserver_port=0 --tserver_master_addrs=127.0.57.254:32871 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.0.57.196 --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.NonVoterReplicasInConsensusQueue.1557210500371863-231/raft_consensus-itest-cluster/ts-3/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.10.0-cdh6.x-SNAPSHOT revision 7d0ee26697926e87af9ef395008799577f6ab43f build type FASTDEBUG built by jenkins at 06 May 2019 22:44:21 PST on quasar-knmpfx-1.vpc.cloudera.com build id 322017 ASAN enabled I0507 06:32:22.158159 9911 system_ntp.cc:202] NTP initialized. Skew: 500ppm Current error: 145516us I0507 06:32:22.158764 9911 fs_manager.cc:263] Metadata directory not provided I0507 06:32:22.158838 9911 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0507 06:32:22.159092 9911 server_base.cc:433] Could not load existing FS layout: Not found: could not find a healthy instance file I0507 06:32:22.159165 9911 server_base.cc:434] Attempting to create new FS layout instead I0507 06:32:22.161574 9911 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.NonVoterReplicasInConsensusQueue.1557210500371863-231/raft_consensus-itest-cluster/ts-3/data/instance: uuid: "0390998780894750a0e7904a91af1410" format_stamp: "Formatted at 2019-05-07 06:32:22 on dist-test-slave-dist-test-slave-rqp6" I0507 06:32:22.162135 9911 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.NonVoterReplicasInConsensusQueue.1557210500371863-231/raft_consensus-itest-cluster/ts-3/wal/instance: uuid: "0390998780894750a0e7904a91af1410" format_stamp: "Formatted at 2019-05-07 06:32:22 on dist-test-slave-dist-test-slave-rqp6" I0507 06:32:22.168872 9911 fs_manager.cc:503] Time spent creating directory manager: real 0.007s user 0.002s sys 0.004s I0507 06:32:22.169891 9911 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0507 06:32:22.170444 9911 file_cache.cc:466] Constructed file cache lbm with capacity 419430 I0507 06:32:22.174022 9911 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.000s sys 0.000s I0507 06:32:22.174093 9911 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.NonVoterReplicasInConsensusQueue.1557210500371863-231/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.NonVoterReplicasInConsensusQueue.1557210500371863-231/raft_consensus-itest-cluster/ts-3/wal uuid: "0390998780894750a0e7904a91af1410" format_stamp: "Formatted at 2019-05-07 06:32:22 on dist-test-slave-dist-test-slave-rqp6" I0507 06:32:22.174253 9911 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.NonVoterReplicasInConsensusQueue.1557210500371863-231/raft_consensus-itest-cluster/ts-3/wal metadata directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.NonVoterReplicasInConsensusQueue.1557210500371863-231/raft_consensus-itest-cluster/ts-3/wal 1 data directories: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.NonVoterReplicasInConsensusQueue.1557210500371863-231/raft_consensus-itest-cluster/ts-3/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0507 06:32:22.222355 9911 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0507 06:32:22.223950 9911 ts_tablet_manager.cc:356] Loaded tablet metadata (0 total tablets, 0 live tablets) I0507 06:32:22.224012 9911 ts_tablet_manager.cc:376] Registered 0 tablets I0507 06:32:22.254500 9911 rpc_server.cc:205] RPC server started. Bound to: 127.0.57.196:33671 I0507 06:32:22.255862 9911 webserver.cc:308] Webserver started at http://127.0.57.196:46383/ using document root and password file I0507 06:32:22.256647 9911 server_base.cc:608] Dumped server information to /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.NonVoterReplicasInConsensusQueue.1557210500371863-231/raft_consensus-itest-cluster/ts-3/data/info.pb I0507 06:32:22.261006 231 external_mini_cluster.cc:995] Started /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver as pid 9911 I0507 06:32:22.269832 10030 heartbeater.cc:345] Connected to a master server at 127.0.57.254:32871 I0507 06:32:22.270061 10030 heartbeater.cc:426] Registering TS with master... I0507 06:32:22.270606 10030 heartbeater.cc:475] Master 127.0.57.254:32871 requested a full tablet report, sending... I0507 06:32:22.271744 9499 ts_manager.cc:159] Registered new tserver with Master: 0390998780894750a0e7904a91af1410 (127.0.57.196:33671) I0507 06:32:22.272805 9499 master_service.cc:273] Signed X509 certificate for tserver {username='slave'} at 127.0.57.196:58803 I0507 06:32:22.274109 231 external_mini_cluster.cc:623] 4 TS(s) registered with all masters I0507 06:32:22.298491 9499 catalog_manager.cc:1415] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44164: name: "TestTable" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 3 split_rows_range_bounds { } partition_schema { range_schema { columns { name: "key" } } } I0507 06:32:22.330473 9600 tablet_service.cc:1016] Processing CreateTablet for tablet 5ae09859764b4956baeac1f4c5241aed (table=TestTable [id=143b7b652f5b4062b9f006416e7db1eb]), partition=RANGE (key) PARTITION UNBOUNDED I0507 06:32:22.331121 9723 tablet_service.cc:1016] Processing CreateTablet for tablet 5ae09859764b4956baeac1f4c5241aed (table=TestTable [id=143b7b652f5b4062b9f006416e7db1eb]), partition=RANGE (key) PARTITION UNBOUNDED I0507 06:32:22.331542 9600 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet 5ae09859764b4956baeac1f4c5241aed. 1 dirs total, 0 dirs full, 0 dirs failed I0507 06:32:22.331959 9723 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet 5ae09859764b4956baeac1f4c5241aed. 1 dirs total, 0 dirs full, 0 dirs failed I0507 06:32:22.341042 10050 tablet_bootstrap.cc:439] T 5ae09859764b4956baeac1f4c5241aed P 0198be74df5a45dea84591d6b6c20c66: Bootstrap starting. I0507 06:32:22.343240 10050 tablet_bootstrap.cc:590] T 5ae09859764b4956baeac1f4c5241aed P 0198be74df5a45dea84591d6b6c20c66: No blocks or log segments found. Creating new log. I0507 06:32:22.343641 9967 tablet_service.cc:1016] Processing CreateTablet for tablet 5ae09859764b4956baeac1f4c5241aed (table=TestTable [id=143b7b652f5b4062b9f006416e7db1eb]), partition=RANGE (key) PARTITION UNBOUNDED I0507 06:32:22.344089 10050 log.cc:525] T 5ae09859764b4956baeac1f4c5241aed P 0198be74df5a45dea84591d6b6c20c66: Log is configured to *not* fsync() on all Append() calls I0507 06:32:22.344594 9967 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet 5ae09859764b4956baeac1f4c5241aed. 1 dirs total, 0 dirs full, 0 dirs failed I0507 06:32:22.345185 10051 tablet_bootstrap.cc:439] T 5ae09859764b4956baeac1f4c5241aed P b46048beca56499c8ee771446de3cbd4: Bootstrap starting. I0507 06:32:22.347313 10051 tablet_bootstrap.cc:590] T 5ae09859764b4956baeac1f4c5241aed P b46048beca56499c8ee771446de3cbd4: No blocks or log segments found. Creating new log. I0507 06:32:22.348134 10051 log.cc:525] T 5ae09859764b4956baeac1f4c5241aed P b46048beca56499c8ee771446de3cbd4: Log is configured to *not* fsync() on all Append() calls I0507 06:32:22.350049 10050 tablet_bootstrap.cc:439] T 5ae09859764b4956baeac1f4c5241aed P 0198be74df5a45dea84591d6b6c20c66: No bootstrap required, opened a new log I0507 06:32:22.350299 10050 ts_tablet_manager.cc:1053] T 5ae09859764b4956baeac1f4c5241aed P 0198be74df5a45dea84591d6b6c20c66: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.000s I0507 06:32:22.353667 10051 tablet_bootstrap.cc:439] T 5ae09859764b4956baeac1f4c5241aed P b46048beca56499c8ee771446de3cbd4: No bootstrap required, opened a new log I0507 06:32:22.353962 10051 ts_tablet_manager.cc:1053] T 5ae09859764b4956baeac1f4c5241aed P b46048beca56499c8ee771446de3cbd4: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.000s I0507 06:32:22.355538 10050 raft_consensus.cc:340] T 5ae09859764b4956baeac1f4c5241aed P 0198be74df5a45dea84591d6b6c20c66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b46048beca56499c8ee771446de3cbd4" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 46277 } } peers { permanent_uuid: "0198be74df5a45dea84591d6b6c20c66" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 40207 } } peers { permanent_uuid: "0390998780894750a0e7904a91af1410" member_type: VOTER last_known_addr { host: "127.0.57.196" port: 33671 } } I0507 06:32:22.355831 10050 raft_consensus.cc:366] T 5ae09859764b4956baeac1f4c5241aed P 0198be74df5a45dea84591d6b6c20c66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0507 06:32:22.355870 10050 raft_consensus.cc:705] T 5ae09859764b4956baeac1f4c5241aed P 0198be74df5a45dea84591d6b6c20c66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0198be74df5a45dea84591d6b6c20c66, State: Initialized, Role: FOLLOWER I0507 06:32:22.356762 10054 tablet_bootstrap.cc:439] T 5ae09859764b4956baeac1f4c5241aed P 0390998780894750a0e7904a91af1410: Bootstrap starting. I0507 06:32:22.358969 10054 tablet_bootstrap.cc:590] T 5ae09859764b4956baeac1f4c5241aed P 0390998780894750a0e7904a91af1410: No blocks or log segments found. Creating new log. I0507 06:32:22.359793 10054 log.cc:525] T 5ae09859764b4956baeac1f4c5241aed P 0390998780894750a0e7904a91af1410: Log is configured to *not* fsync() on all Append() calls I0507 06:32:22.360332 10050 consensus_queue.cc:229] T 5ae09859764b4956baeac1f4c5241aed P 0198be74df5a45dea84591d6b6c20c66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b46048beca56499c8ee771446de3cbd4" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 46277 } } peers { permanent_uuid: "0198be74df5a45dea84591d6b6c20c66" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 40207 } } peers { permanent_uuid: "0390998780894750a0e7904a91af1410" member_type: VOTER last_known_addr { host: "127.0.57.196" port: 33671 } } I0507 06:32:22.361232 9786 heartbeater.cc:467] Master 127.0.57.254:32871 was elected leader, sending a full tablet report... I0507 06:32:22.361013 10051 raft_consensus.cc:340] T 5ae09859764b4956baeac1f4c5241aed P b46048beca56499c8ee771446de3cbd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b46048beca56499c8ee771446de3cbd4" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 46277 } } peers { permanent_uuid: "0198be74df5a45dea84591d6b6c20c66" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 40207 } } peers { permanent_uuid: "0390998780894750a0e7904a91af1410" member_type: VOTER last_known_addr { host: "127.0.57.196" port: 33671 } } I0507 06:32:22.361331 10051 raft_consensus.cc:366] T 5ae09859764b4956baeac1f4c5241aed P b46048beca56499c8ee771446de3cbd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0507 06:32:22.361387 10051 raft_consensus.cc:705] T 5ae09859764b4956baeac1f4c5241aed P b46048beca56499c8ee771446de3cbd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b46048beca56499c8ee771446de3cbd4, State: Initialized, Role: FOLLOWER I0507 06:32:22.361815 10051 consensus_queue.cc:229] T 5ae09859764b4956baeac1f4c5241aed P b46048beca56499c8ee771446de3cbd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b46048beca56499c8ee771446de3cbd4" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 46277 } } peers { permanent_uuid: "0198be74df5a45dea84591d6b6c20c66" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 40207 } } peers { permanent_uuid: "0390998780894750a0e7904a91af1410" member_type: VOTER last_known_addr { host: "127.0.57.196" port: 33671 } } I0507 06:32:22.362072 10054 tablet_bootstrap.cc:439] T 5ae09859764b4956baeac1f4c5241aed P 0390998780894750a0e7904a91af1410: No bootstrap required, opened a new log I0507 06:32:22.362314 10054 ts_tablet_manager.cc:1053] T 5ae09859764b4956baeac1f4c5241aed P 0390998780894750a0e7904a91af1410: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s I0507 06:32:22.364920 9663 heartbeater.cc:467] Master 127.0.57.254:32871 was elected leader, sending a full tablet report... I0507 06:32:22.365296 10051 ts_tablet_manager.cc:1081] T 5ae09859764b4956baeac1f4c5241aed P b46048beca56499c8ee771446de3cbd4: Time spent starting tablet: real 0.011s user 0.001s sys 0.009s I0507 06:32:22.365406 10050 ts_tablet_manager.cc:1081] T 5ae09859764b4956baeac1f4c5241aed P 0198be74df5a45dea84591d6b6c20c66: Time spent starting tablet: real 0.015s user 0.007s sys 0.000s I0507 06:32:22.369156 10054 raft_consensus.cc:340] T 5ae09859764b4956baeac1f4c5241aed P 0390998780894750a0e7904a91af1410 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b46048beca56499c8ee771446de3cbd4" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 46277 } } peers { permanent_uuid: "0198be74df5a45dea84591d6b6c20c66" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 40207 } } peers { permanent_uuid: "0390998780894750a0e7904a91af1410" member_type: VOTER last_known_addr { host: "127.0.57.196" port: 33671 } } I0507 06:32:22.369459 10054 raft_consensus.cc:366] T 5ae09859764b4956baeac1f4c5241aed P 0390998780894750a0e7904a91af1410 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0507 06:32:22.369513 10054 raft_consensus.cc:705] T 5ae09859764b4956baeac1f4c5241aed P 0390998780894750a0e7904a91af1410 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0390998780894750a0e7904a91af1410, State: Initialized, Role: FOLLOWER I0507 06:32:22.369882 10054 consensus_queue.cc:229] T 5ae09859764b4956baeac1f4c5241aed P 0390998780894750a0e7904a91af1410 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b46048beca56499c8ee771446de3cbd4" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 46277 } } peers { permanent_uuid: "0198be74df5a45dea84591d6b6c20c66" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 40207 } } peers { permanent_uuid: "0390998780894750a0e7904a91af1410" member_type: VOTER last_known_addr { host: "127.0.57.196" port: 33671 } } I0507 06:32:22.370600 10030 heartbeater.cc:467] Master 127.0.57.254:32871 was elected leader, sending a full tablet report... I0507 06:32:22.370960 10054 ts_tablet_manager.cc:1081] T 5ae09859764b4956baeac1f4c5241aed P 0390998780894750a0e7904a91af1410: Time spent starting tablet: real 0.009s user 0.006s sys 0.000s I0507 06:32:22.379415 10058 raft_consensus.cc:472] T 5ae09859764b4956baeac1f4c5241aed P 0390998780894750a0e7904a91af1410 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0507 06:32:22.379575 10058 raft_consensus.cc:494] T 5ae09859764b4956baeac1f4c5241aed P 0390998780894750a0e7904a91af1410 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b46048beca56499c8ee771446de3cbd4" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 46277 } } peers { permanent_uuid: "0198be74df5a45dea84591d6b6c20c66" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 40207 } } peers { permanent_uuid: "0390998780894750a0e7904a91af1410" member_type: VOTER last_known_addr { host: "127.0.57.196" port: 33671 } } I0507 06:32:22.380715 10058 leader_election.cc:296] T 5ae09859764b4956baeac1f4c5241aed P 0390998780894750a0e7904a91af1410 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b46048beca56499c8ee771446de3cbd4 (127.0.57.193:46277), 0198be74df5a45dea84591d6b6c20c66 (127.0.57.194:40207) I0507 06:32:22.385742 9743 tablet_service.cc:1286] Received RequestConsensusVote() RPC: tablet_id: "5ae09859764b4956baeac1f4c5241aed" candidate_uuid: "0390998780894750a0e7904a91af1410" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0198be74df5a45dea84591d6b6c20c66" is_pre_election: true I0507 06:32:22.386124 9743 raft_consensus.cc:2346] T 5ae09859764b4956baeac1f4c5241aed P 0198be74df5a45dea84591d6b6c20c66 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0390998780894750a0e7904a91af1410 in term 0. I0507 06:32:22.386799 9918 leader_election.cc:310] T 5ae09859764b4956baeac1f4c5241aed P 0390998780894750a0e7904a91af1410 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0198be74df5a45dea84591d6b6c20c66, 0390998780894750a0e7904a91af1410; no voters: I0507 06:32:22.387367 10058 raft_consensus.cc:2640] T 5ae09859764b4956baeac1f4c5241aed P 0390998780894750a0e7904a91af1410 [term 0 FOLLOWER]: Leader pre-election won for term 1 I0507 06:32:22.387465 10058 raft_consensus.cc:472] T 5ae09859764b4956baeac1f4c5241aed P 0390998780894750a0e7904a91af1410 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0507 06:32:22.387519 10058 raft_consensus.cc:2886] T 5ae09859764b4956baeac1f4c5241aed P 0390998780894750a0e7904a91af1410 [term 0 FOLLOWER]: Advancing to term 1 I0507 06:32:22.389343 10058 raft_consensus.cc:494] T 5ae09859764b4956baeac1f4c5241aed P 0390998780894750a0e7904a91af1410 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b46048beca56499c8ee771446de3cbd4" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 46277 } } peers { permanent_uuid: "0198be74df5a45dea84591d6b6c20c66" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 40207 } } peers { permanent_uuid: "0390998780894750a0e7904a91af1410" member_type: VOTER last_known_addr { host: "127.0.57.196" port: 33671 } } I0507 06:32:22.389952 10058 leader_election.cc:296] T 5ae09859764b4956baeac1f4c5241aed P 0390998780894750a0e7904a91af1410 [CANDIDATE]: Term 1 election: Requested vote from peers b46048beca56499c8ee771446de3cbd4 (127.0.57.193:46277), 0198be74df5a45dea84591d6b6c20c66 (127.0.57.194:40207) I0507 06:32:22.390290 9743 tablet_service.cc:1286] Received RequestConsensusVote() RPC: tablet_id: "5ae09859764b4956baeac1f4c5241aed" candidate_uuid: "0390998780894750a0e7904a91af1410" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0198be74df5a45dea84591d6b6c20c66" I0507 06:32:22.390321 9619 tablet_service.cc:1286] Received RequestConsensusVote() RPC: tablet_id: "5ae09859764b4956baeac1f4c5241aed" candidate_uuid: "0390998780894750a0e7904a91af1410" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b46048beca56499c8ee771446de3cbd4" I0507 06:32:22.390166 9620 tablet_service.cc:1286] Received RequestConsensusVote() RPC: tablet_id: "5ae09859764b4956baeac1f4c5241aed" candidate_uuid: "0390998780894750a0e7904a91af1410" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b46048beca56499c8ee771446de3cbd4" is_pre_election: true I0507 06:32:22.390450 9743 raft_consensus.cc:2886] T 5ae09859764b4956baeac1f4c5241aed P 0198be74df5a45dea84591d6b6c20c66 [term 0 FOLLOWER]: Advancing to term 1 I0507 06:32:22.390579 9619 raft_consensus.cc:2886] T 5ae09859764b4956baeac1f4c5241aed P b46048beca56499c8ee771446de3cbd4 [term 0 FOLLOWER]: Advancing to term 1 I0507 06:32:22.392305 9619 raft_consensus.cc:2346] T 5ae09859764b4956baeac1f4c5241aed P b46048beca56499c8ee771446de3cbd4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0390998780894750a0e7904a91af1410 in term 1. I0507 06:32:22.392400 9743 raft_consensus.cc:2346] T 5ae09859764b4956baeac1f4c5241aed P 0198be74df5a45dea84591d6b6c20c66 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0390998780894750a0e7904a91af1410 in term 1. I0507 06:32:22.392516 9620 raft_consensus.cc:2321] T 5ae09859764b4956baeac1f4c5241aed P b46048beca56499c8ee771446de3cbd4 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0390998780894750a0e7904a91af1410 for term 1 because replica is already servicing an update from a current leader or another vote. I0507 06:32:22.392833 9918 leader_election.cc:310] T 5ae09859764b4956baeac1f4c5241aed P 0390998780894750a0e7904a91af1410 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0198be74df5a45dea84591d6b6c20c66, 0390998780894750a0e7904a91af1410; no voters: I0507 06:32:22.393123 10058 raft_consensus.cc:2640] T 5ae09859764b4956baeac1f4c5241aed P 0390998780894750a0e7904a91af1410 [term 1 FOLLOWER]: Leader election won for term 1 I0507 06:32:22.393561 10058 raft_consensus.cc:667] T 5ae09859764b4956baeac1f4c5241aed P 0390998780894750a0e7904a91af1410 [term 1 LEADER]: Becoming Leader. State: Replica: 0390998780894750a0e7904a91af1410, State: Running, Role: LEADER I0507 06:32:22.393915 10058 consensus_queue.cc:206] T 5ae09859764b4956baeac1f4c5241aed P 0390998780894750a0e7904a91af1410 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b46048beca56499c8ee771446de3cbd4" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 46277 } } peers { permanent_uuid: "0198be74df5a45dea84591d6b6c20c66" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 40207 } } peers { permanent_uuid: "0390998780894750a0e7904a91af1410" member_type: VOTER last_known_addr { host: "127.0.57.196" port: 33671 } } I0507 06:32:22.398245 9499 catalog_manager.cc:3988] T 5ae09859764b4956baeac1f4c5241aed P 0390998780894750a0e7904a91af1410 reported cstate change: term changed from 0 to 1, leader changed from to 0390998780894750a0e7904a91af1410 (127.0.57.196). New cstate: current_term: 1 leader_uuid: "0390998780894750a0e7904a91af1410" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b46048beca56499c8ee771446de3cbd4" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 46277 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0198be74df5a45dea84591d6b6c20c66" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 40207 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0390998780894750a0e7904a91af1410" member_type: VOTER last_known_addr { host: "127.0.57.196" port: 33671 } health_report { overall_health: HEALTHY } } } I0507 06:32:22.423413 231 external_mini_cluster.cc:623] 4 TS(s) registered with all masters I0507 06:32:22.424798 231 ts_itest-base.cc:232] Waiting for 1 tablets on tserver b46048beca56499c8ee771446de3cbd4 to finish bootstrapping I0507 06:32:22.431291 231 ts_itest-base.cc:232] Waiting for 1 tablets on tserver 0198be74df5a45dea84591d6b6c20c66 to finish bootstrapping I0507 06:32:22.437606 9620 raft_consensus.cc:1181] T 5ae09859764b4956baeac1f4c5241aed P b46048beca56499c8ee771446de3cbd4 [term 1 FOLLOWER]: Refusing update from remote peer 0390998780894750a0e7904a91af1410: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch) I0507 06:32:22.437664 231 ts_itest-base.cc:232] Waiting for 0 tablets on tserver a6726eec155e4dc48c6ae00643563ab4 to finish bootstrapping I0507 06:32:22.438491 10060 consensus_queue.cc:984] T 5ae09859764b4956baeac1f4c5241aed P 0390998780894750a0e7904a91af1410 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b46048beca56499c8ee771446de3cbd4" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 46277 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0507 06:32:22.439245 9743 raft_consensus.cc:1181] T 5ae09859764b4956baeac1f4c5241aed P 0198be74df5a45dea84591d6b6c20c66 [term 1 FOLLOWER]: Refusing update from remote peer 0390998780894750a0e7904a91af1410: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch) I0507 06:32:22.439906 10060 consensus_queue.cc:984] T 5ae09859764b4956baeac1f4c5241aed P 0390998780894750a0e7904a91af1410 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0198be74df5a45dea84591d6b6c20c66" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 40207 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0507 06:32:22.446868 231 ts_itest-base.cc:232] Waiting for 1 tablets on tserver 0390998780894750a0e7904a91af1410 to finish bootstrapping W0507 06:32:22.468492 9660 generic_service.cc:128] {username='slave'} at 127.0.0.1:54330 forcing change of non-runtime-safe flag enable_leader_failure_detection I0507 06:32:22.468653 9660 generic_service.cc:148] {username='slave'} at 127.0.0.1:54330 changed flags via RPC: enable_leader_failure_detection from 'true' to 'false' W0507 06:32:22.469885 9783 generic_service.cc:128] {username='slave'} at 127.0.0.1:57892 forcing change of non-runtime-safe flag enable_leader_failure_detection I0507 06:32:22.470039 9783 generic_service.cc:148] {username='slave'} at 127.0.0.1:57892 changed flags via RPC: enable_leader_failure_detection from 'true' to 'false' W0507 06:32:22.471442 9905 generic_service.cc:128] {username='slave'} at 127.0.0.1:46000 forcing change of non-runtime-safe flag enable_leader_failure_detection I0507 06:32:22.471560 9905 generic_service.cc:148] {username='slave'} at 127.0.0.1:46000 changed flags via RPC: enable_leader_failure_detection from 'true' to 'false' W0507 06:32:22.472798 10027 generic_service.cc:128] {username='slave'} at 127.0.0.1:40748 forcing change of non-runtime-safe flag enable_leader_failure_detection I0507 06:32:22.472949 10027 generic_service.cc:148] {username='slave'} at 127.0.0.1:40748 changed flags via RPC: enable_leader_failure_detection from 'true' to 'false' I0507 06:32:22.473501 231 raft_consensus_nonvoter-itest.cc:1803] Getting leader replica... I0507 06:32:22.476413 231 raft_consensus_nonvoter-itest.cc:1807] Shutting down non-leader replicas... I0507 06:32:22.476492 231 external_mini_cluster.cc:1172] Attempting to check leaks for /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver pid 9533 I0507 06:32:22.477167 9660 generic_service.cc:171] Checking for leaks (request via RPC) I0507 06:32:22.527447 231 external_mini_cluster.cc:1131] Killing /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver with pid 9533 I0507 06:32:22.539626 231 external_mini_cluster.cc:1172] Attempting to check leaks for /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver pid 9667 I0507 06:32:22.540685 9783 generic_service.cc:171] Checking for leaks (request via RPC) W0507 06:32:22.541568 9920 consensus_peers.cc:457] T 5ae09859764b4956baeac1f4c5241aed P 0390998780894750a0e7904a91af1410 -> Peer b46048beca56499c8ee771446de3cbd4 (127.0.57.193:46277): Couldn't send request to peer b46048beca56499c8ee771446de3cbd4. Status: Network error: Client connection negotiation failed: client connection to 127.0.57.193:46277: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry. I0507 06:32:22.589846 231 external_mini_cluster.cc:1131] Killing /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver with pid 9667 I0507 06:32:22.601490 231 raft_consensus_nonvoter-itest.cc:1814] Adding NON_VOTER replica... I0507 06:32:22.601922 231 raft_consensus_nonvoter-itest.cc:1821] Waiting for pending config... I0507 06:32:22.604233 9986 consensus_queue.cc:206] T 5ae09859764b4956baeac1f4c5241aed P 0390998780894750a0e7904a91af1410 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b46048beca56499c8ee771446de3cbd4" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 46277 } } peers { permanent_uuid: "0198be74df5a45dea84591d6b6c20c66" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 40207 } } peers { permanent_uuid: "0390998780894750a0e7904a91af1410" member_type: VOTER last_known_addr { host: "127.0.57.196" port: 33671 } } peers { permanent_uuid: "a6726eec155e4dc48c6ae00643563ab4" member_type: NON_VOTER last_known_addr { host: "127.0.57.195" port: 38581 } attrs { } } W0507 06:32:22.606485 9920 consensus_peers.cc:457] T 5ae09859764b4956baeac1f4c5241aed P 0390998780894750a0e7904a91af1410 -> Peer b46048beca56499c8ee771446de3cbd4 (127.0.57.193:46277): Couldn't send request to peer b46048beca56499c8ee771446de3cbd4. Status: Network error: Client connection negotiation failed: client connection to 127.0.57.193:46277: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry. W0507 06:32:22.606891 9918 consensus_peers.cc:457] T 5ae09859764b4956baeac1f4c5241aed P 0390998780894750a0e7904a91af1410 -> Peer 0198be74df5a45dea84591d6b6c20c66 (127.0.57.194:40207): Couldn't send request to peer 0198be74df5a45dea84591d6b6c20c66. Status: Network error: Client connection negotiation failed: client connection to 127.0.57.194:40207: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry. W0507 06:32:22.612082 9921 consensus_peers.cc:457] T 5ae09859764b4956baeac1f4c5241aed P 0390998780894750a0e7904a91af1410 -> Peer a6726eec155e4dc48c6ae00643563ab4 (127.0.57.195:38581): Couldn't send request to peer a6726eec155e4dc48c6ae00643563ab4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5ae09859764b4956baeac1f4c5241aed. This is attempt 1: this message will repeat every 5th retry. I0507 06:32:22.666811 10076 ts_tablet_manager.cc:676] T 5ae09859764b4956baeac1f4c5241aed P a6726eec155e4dc48c6ae00643563ab4: Initiating tablet copy from peer 0390998780894750a0e7904a91af1410 (127.0.57.196:33671) I0507 06:32:22.667172 10076 tablet_copy_client.cc:240] T 5ae09859764b4956baeac1f4c5241aed P a6726eec155e4dc48c6ae00643563ab4: tablet copy: Beginning tablet copy session from remote peer at address 127.0.57.196:33671 I0507 06:32:22.672830 10007 tablet_copy_service.cc:135] P 0390998780894750a0e7904a91af1410: Received BeginTabletCopySession request for tablet 5ae09859764b4956baeac1f4c5241aed from peer a6726eec155e4dc48c6ae00643563ab4 ({username='slave'} at 127.0.57.195:47541) I0507 06:32:22.673038 10007 tablet_copy_service.cc:156] P 0390998780894750a0e7904a91af1410: Beginning new tablet copy session on tablet 5ae09859764b4956baeac1f4c5241aed from peer a6726eec155e4dc48c6ae00643563ab4 at {username='slave'} at 127.0.57.195:47541: session id = a6726eec155e4dc48c6ae00643563ab4-5ae09859764b4956baeac1f4c5241aed I0507 06:32:22.676017 10007 tablet_copy_source_session.cc:202] T 5ae09859764b4956baeac1f4c5241aed P 0390998780894750a0e7904a91af1410: Tablet Copy: opened 0 blocks and 1 log segments I0507 06:32:22.679769 10076 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet 5ae09859764b4956baeac1f4c5241aed. 1 dirs total, 0 dirs full, 0 dirs failed I0507 06:32:22.688446 10076 tablet_copy_client.cc:575] T 5ae09859764b4956baeac1f4c5241aed P a6726eec155e4dc48c6ae00643563ab4: tablet copy: Starting download of 0 data blocks... I0507 06:32:22.688637 10076 tablet_copy_client.cc:538] T 5ae09859764b4956baeac1f4c5241aed P a6726eec155e4dc48c6ae00643563ab4: tablet copy: Starting download of 1 WAL segments... I0507 06:32:22.690593 10076 tablet_copy_client.cc:414] T 5ae09859764b4956baeac1f4c5241aed P a6726eec155e4dc48c6ae00643563ab4: tablet copy: Tablet Copy complete. Replacing tablet superblock. I0507 06:32:22.692965 10076 tablet_bootstrap.cc:439] T 5ae09859764b4956baeac1f4c5241aed P a6726eec155e4dc48c6ae00643563ab4: Bootstrap starting. I0507 06:32:22.697726 10076 log.cc:525] T 5ae09859764b4956baeac1f4c5241aed P a6726eec155e4dc48c6ae00643563ab4: Log is configured to *not* fsync() on all Append() calls I0507 06:32:22.700328 10076 tablet_bootstrap.cc:439] T 5ae09859764b4956baeac1f4c5241aed P a6726eec155e4dc48c6ae00643563ab4: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates I0507 06:32:22.700718 10076 tablet_bootstrap.cc:439] T 5ae09859764b4956baeac1f4c5241aed P a6726eec155e4dc48c6ae00643563ab4: Bootstrap complete. I0507 06:32:22.700999 10076 ts_tablet_manager.cc:1053] T 5ae09859764b4956baeac1f4c5241aed P a6726eec155e4dc48c6ae00643563ab4: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s I0507 06:32:22.706130 10076 raft_consensus.cc:340] T 5ae09859764b4956baeac1f4c5241aed P a6726eec155e4dc48c6ae00643563ab4 [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b46048beca56499c8ee771446de3cbd4" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 46277 } } peers { permanent_uuid: "0198be74df5a45dea84591d6b6c20c66" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 40207 } } peers { permanent_uuid: "0390998780894750a0e7904a91af1410" member_type: VOTER last_known_addr { host: "127.0.57.196" port: 33671 } } I0507 06:32:22.706990 10076 raft_consensus.cc:705] T 5ae09859764b4956baeac1f4c5241aed P a6726eec155e4dc48c6ae00643563ab4 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a6726eec155e4dc48c6ae00643563ab4, State: Initialized, Role: LEARNER I0507 06:32:22.707252 10076 consensus_queue.cc:229] T 5ae09859764b4956baeac1f4c5241aed P a6726eec155e4dc48c6ae00643563ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b46048beca56499c8ee771446de3cbd4" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 46277 } } peers { permanent_uuid: "0198be74df5a45dea84591d6b6c20c66" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 40207 } } peers { permanent_uuid: "0390998780894750a0e7904a91af1410" member_type: VOTER last_known_addr { host: "127.0.57.196" port: 33671 } } peers { permanent_uuid: "a6726eec155e4dc48c6ae00643563ab4" member_type: NON_VOTER last_known_addr { host: "127.0.57.195" port: 38581 } attrs { } } I0507 06:32:22.708042 9908 heartbeater.cc:467] Master 127.0.57.254:32871 was elected leader, sending a full tablet report... I0507 06:32:22.708597 10076 ts_tablet_manager.cc:1081] T 5ae09859764b4956baeac1f4c5241aed P a6726eec155e4dc48c6ae00643563ab4: Time spent starting tablet: real 0.008s user 0.003s sys 0.004s I0507 06:32:22.709336 10007 tablet_copy_service.cc:325] P 0390998780894750a0e7904a91af1410: Request end of tablet copy session a6726eec155e4dc48c6ae00643563ab4-5ae09859764b4956baeac1f4c5241aed received from {username='slave'} at 127.0.57.195:47541 I0507 06:32:22.709446 10007 tablet_copy_service.cc:413] P 0390998780894750a0e7904a91af1410: ending tablet copy session a6726eec155e4dc48c6ae00643563ab4-5ae09859764b4956baeac1f4c5241aed on tablet 5ae09859764b4956baeac1f4c5241aed with peer a6726eec155e4dc48c6ae00643563ab4 I0507 06:32:22.747840 9865 raft_consensus.cc:1126] T 5ae09859764b4956baeac1f4c5241aed P a6726eec155e4dc48c6ae00643563ab4 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[] W0507 06:32:22.837538 9920 consensus_peers.cc:457] T 5ae09859764b4956baeac1f4c5241aed P 0390998780894750a0e7904a91af1410 -> Peer b46048beca56499c8ee771446de3cbd4 (127.0.57.193:46277): Couldn't send request to peer b46048beca56499c8ee771446de3cbd4. Status: Network error: Client connection negotiation failed: client connection to 127.0.57.193:46277: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry. W0507 06:32:22.840461 9918 consensus_peers.cc:457] T 5ae09859764b4956baeac1f4c5241aed P 0390998780894750a0e7904a91af1410 -> Peer 0198be74df5a45dea84591d6b6c20c66 (127.0.57.194:40207): Couldn't send request to peer 0198be74df5a45dea84591d6b6c20c66. Status: Network error: Client connection negotiation failed: client connection to 127.0.57.194:40207: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry. W0507 06:32:23.090574 9918 consensus_peers.cc:457] T 5ae09859764b4956baeac1f4c5241aed P 0390998780894750a0e7904a91af1410 -> Peer 0198be74df5a45dea84591d6b6c20c66 (127.0.57.194:40207): Couldn't send request to peer 0198be74df5a45dea84591d6b6c20c66. Status: Network error: Client connection negotiation failed: client connection to 127.0.57.194:40207: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry. W0507 06:32:23.118525 9920 consensus_peers.cc:457] T 5ae09859764b4956baeac1f4c5241aed P 0390998780894750a0e7904a91af1410 -> Peer b46048beca56499c8ee771446de3cbd4 (127.0.57.193:46277): Couldn't send request to peer b46048beca56499c8ee771446de3cbd4. Status: Network error: Client connection negotiation failed: client connection to 127.0.57.193:46277: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry. W0507 06:32:23.345634 9918 consensus_peers.cc:457] T 5ae09859764b4956baeac1f4c5241aed P 0390998780894750a0e7904a91af1410 -> Peer 0198be74df5a45dea84591d6b6c20c66 (127.0.57.194:40207): Couldn't send request to peer 0198be74df5a45dea84591d6b6c20c66. Status: Network error: Client connection negotiation failed: client connection to 127.0.57.194:40207: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry. W0507 06:32:23.382936 9920 consensus_peers.cc:457] T 5ae09859764b4956baeac1f4c5241aed P 0390998780894750a0e7904a91af1410 -> Peer b46048beca56499c8ee771446de3cbd4 (127.0.57.193:46277): Couldn't send request to peer b46048beca56499c8ee771446de3cbd4. Status: Network error: Client connection negotiation failed: client connection to 127.0.57.193:46277: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry. W0507 06:32:23.582195 9918 consensus_peers.cc:457] T 5ae09859764b4956baeac1f4c5241aed P 0390998780894750a0e7904a91af1410 -> Peer 0198be74df5a45dea84591d6b6c20c66 (127.0.57.194:40207): Couldn't send request to peer 0198be74df5a45dea84591d6b6c20c66. Status: Network error: Client connection negotiation failed: client connection to 127.0.57.194:40207: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry. W0507 06:32:23.635581 9920 consensus_peers.cc:457] T 5ae09859764b4956baeac1f4c5241aed P 0390998780894750a0e7904a91af1410 -> Peer b46048beca56499c8ee771446de3cbd4 (127.0.57.193:46277): Couldn't send request to peer b46048beca56499c8ee771446de3cbd4. Status: Network error: Client connection negotiation failed: client connection to 127.0.57.193:46277: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry. W0507 06:32:23.816609 9918 consensus_peers.cc:457] T 5ae09859764b4956baeac1f4c5241aed P 0390998780894750a0e7904a91af1410 -> Peer 0198be74df5a45dea84591d6b6c20c66 (127.0.57.194:40207): Couldn't send request to peer 0198be74df5a45dea84591d6b6c20c66. Status: Network error: Client connection negotiation failed: client connection to 127.0.57.194:40207: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry. W0507 06:32:23.874315 9920 consensus_peers.cc:457] T 5ae09859764b4956baeac1f4c5241aed P 0390998780894750a0e7904a91af1410 -> Peer b46048beca56499c8ee771446de3cbd4 (127.0.57.193:46277): Couldn't send request to peer b46048beca56499c8ee771446de3cbd4. Status: Network error: Client connection negotiation failed: client connection to 127.0.57.193:46277: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry. W0507 06:32:24.096551 9918 consensus_peers.cc:457] T 5ae09859764b4956baeac1f4c5241aed P 0390998780894750a0e7904a91af1410 -> Peer 0198be74df5a45dea84591d6b6c20c66 (127.0.57.194:40207): Couldn't send request to peer 0198be74df5a45dea84591d6b6c20c66. Status: Network error: Client connection negotiation failed: client connection to 127.0.57.194:40207: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry. W0507 06:32:24.125825 9920 consensus_peers.cc:457] T 5ae09859764b4956baeac1f4c5241aed P 0390998780894750a0e7904a91af1410 -> Peer b46048beca56499c8ee771446de3cbd4 (127.0.57.193:46277): Couldn't send request to peer b46048beca56499c8ee771446de3cbd4. Status: Network error: Client connection negotiation failed: client connection to 127.0.57.193:46277: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry. W0507 06:32:24.330786 9918 consensus_peers.cc:457] T 5ae09859764b4956baeac1f4c5241aed P 0390998780894750a0e7904a91af1410 -> Peer 0198be74df5a45dea84591d6b6c20c66 (127.0.57.194:40207): Couldn't send request to peer 0198be74df5a45dea84591d6b6c20c66. Status: Network error: Client connection negotiation failed: client connection to 127.0.57.194:40207: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry. W0507 06:32:24.362069 9920 consensus_peers.cc:457] T 5ae09859764b4956baeac1f4c5241aed P 0390998780894750a0e7904a91af1410 -> Peer b46048beca56499c8ee771446de3cbd4 (127.0.57.193:46277): Couldn't send request to peer b46048beca56499c8ee771446de3cbd4. Status: Network error: Client connection negotiation failed: client connection to 127.0.57.193:46277: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry. W0507 06:32:24.582816 9918 consensus_peers.cc:457] T 5ae09859764b4956baeac1f4c5241aed P 0390998780894750a0e7904a91af1410 -> Peer 0198be74df5a45dea84591d6b6c20c66 (127.0.57.194:40207): Couldn't send request to peer 0198be74df5a45dea84591d6b6c20c66. Status: Network error: Client connection negotiation failed: client connection to 127.0.57.194:40207: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry. W0507 06:32:24.621204 9920 consensus_peers.cc:457] T 5ae09859764b4956baeac1f4c5241aed P 0390998780894750a0e7904a91af1410 -> Peer b46048beca56499c8ee771446de3cbd4 (127.0.57.193:46277): Couldn't send request to peer b46048beca56499c8ee771446de3cbd4. Status: Network error: Client connection negotiation failed: client connection to 127.0.57.193:46277: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry. W0507 06:32:24.829289 9918 consensus_peers.cc:457] T 5ae09859764b4956baeac1f4c5241aed P 0390998780894750a0e7904a91af1410 -> Peer 0198be74df5a45dea84591d6b6c20c66 (127.0.57.194:40207): Couldn't send request to peer 0198be74df5a45dea84591d6b6c20c66. Status: Network error: Client connection negotiation failed: client connection to 127.0.57.194:40207: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry. W0507 06:32:24.871898 9920 consensus_peers.cc:457] T 5ae09859764b4956baeac1f4c5241aed P 0390998780894750a0e7904a91af1410 -> Peer b46048beca56499c8ee771446de3cbd4 (127.0.57.193:46277): Couldn't send request to peer b46048beca56499c8ee771446de3cbd4. Status: Network error: Client connection negotiation failed: client connection to 127.0.57.193:46277: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry. W0507 06:32:25.089607 9918 consensus_peers.cc:457] T 5ae09859764b4956baeac1f4c5241aed P 0390998780894750a0e7904a91af1410 -> Peer 0198be74df5a45dea84591d6b6c20c66 (127.0.57.194:40207): Couldn't send request to peer 0198be74df5a45dea84591d6b6c20c66. Status: Network error: Client connection negotiation failed: client connection to 127.0.57.194:40207: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry. W0507 06:32:25.120775 9920 consensus_peers.cc:457] T 5ae09859764b4956baeac1f4c5241aed P 0390998780894750a0e7904a91af1410 -> Peer b46048beca56499c8ee771446de3cbd4 (127.0.57.193:46277): Couldn't send request to peer b46048beca56499c8ee771446de3cbd4. Status: Network error: Client connection negotiation failed: client connection to 127.0.57.193:46277: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry. W0507 06:32:25.329474 9918 consensus_peers.cc:457] T 5ae09859764b4956baeac1f4c5241aed P 0390998780894750a0e7904a91af1410 -> Peer 0198be74df5a45dea84591d6b6c20c66 (127.0.57.194:40207): Couldn't send request to peer 0198be74df5a45dea84591d6b6c20c66. Status: Network error: Client connection negotiation failed: client connection to 127.0.57.194:40207: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry. W0507 06:32:25.379328 9920 consensus_peers.cc:457] T 5ae09859764b4956baeac1f4c5241aed P 0390998780894750a0e7904a91af1410 -> Peer b46048beca56499c8ee771446de3cbd4 (127.0.57.193:46277): Couldn't send request to peer b46048beca56499c8ee771446de3cbd4. Status: Network error: Client connection negotiation failed: client connection to 127.0.57.193:46277: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry. W0507 06:32:25.590557 9918 consensus_peers.cc:457] T 5ae09859764b4956baeac1f4c5241aed P 0390998780894750a0e7904a91af1410 -> Peer 0198be74df5a45dea84591d6b6c20c66 (127.0.57.194:40207): Couldn't send request to peer 0198be74df5a45dea84591d6b6c20c66. Status: Network error: Client connection negotiation failed: client connection to 127.0.57.194:40207: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry. W0507 06:32:25.639430 9920 consensus_peers.cc:457] T 5ae09859764b4956baeac1f4c5241aed P 0390998780894750a0e7904a91af1410 -> Peer b46048beca56499c8ee771446de3cbd4 (127.0.57.193:46277): Couldn't send request to peer b46048beca56499c8ee771446de3cbd4. Status: Network error: Client connection negotiation failed: client connection to 127.0.57.193:46277: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry. W0507 06:32:25.835074 9918 consensus_peers.cc:457] T 5ae09859764b4956baeac1f4c5241aed P 0390998780894750a0e7904a91af1410 -> Peer 0198be74df5a45dea84591d6b6c20c66 (127.0.57.194:40207): Couldn't send request to peer 0198be74df5a45dea84591d6b6c20c66. Status: Network error: Client connection negotiation failed: client connection to 127.0.57.194:40207: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry. W0507 06:32:25.889466 9920 consensus_peers.cc:457] T 5ae09859764b4956baeac1f4c5241aed P 0390998780894750a0e7904a91af1410 -> Peer b46048beca56499c8ee771446de3cbd4 (127.0.57.193:46277): Couldn't send request to peer b46048beca56499c8ee771446de3cbd4. Status: Network error: Client connection negotiation failed: client connection to 127.0.57.193:46277: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry. W0507 06:32:26.091606 9918 consensus_peers.cc:457] T 5ae09859764b4956baeac1f4c5241aed P 0390998780894750a0e7904a91af1410 -> Peer 0198be74df5a45dea84591d6b6c20c66 (127.0.57.194:40207): Couldn't send request to peer 0198be74df5a45dea84591d6b6c20c66. Status: Network error: Client connection negotiation failed: client connection to 127.0.57.194:40207: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry. W0507 06:32:26.143792 9920 consensus_peers.cc:457] T 5ae09859764b4956baeac1f4c5241aed P 0390998780894750a0e7904a91af1410 -> Peer b46048beca56499c8ee771446de3cbd4 (127.0.57.193:46277): Couldn't send request to peer b46048beca56499c8ee771446de3cbd4. Status: Network error: Client connection negotiation failed: client connection to 127.0.57.193:46277: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry. W0507 06:32:26.337517 9918 consensus_peers.cc:457] T 5ae09859764b4956baeac1f4c5241aed P 0390998780894750a0e7904a91af1410 -> Peer 0198be74df5a45dea84591d6b6c20c66 (127.0.57.194:40207): Couldn't send request to peer 0198be74df5a45dea84591d6b6c20c66. Status: Network error: Client connection negotiation failed: client connection to 127.0.57.194:40207: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry. W0507 06:32:26.363715 9920 consensus_peers.cc:457] T 5ae09859764b4956baeac1f4c5241aed P 0390998780894750a0e7904a91af1410 -> Peer b46048beca56499c8ee771446de3cbd4 (127.0.57.193:46277): Couldn't send request to peer b46048beca56499c8ee771446de3cbd4. Status: Network error: Client connection negotiation failed: client connection to 127.0.57.193:46277: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry. W0507 06:32:26.574290 9918 consensus_peers.cc:457] T 5ae09859764b4956baeac1f4c5241aed P 0390998780894750a0e7904a91af1410 -> Peer 0198be74df5a45dea84591d6b6c20c66 (127.0.57.194:40207): Couldn't send request to peer 0198be74df5a45dea84591d6b6c20c66. Status: Network error: Client connection negotiation failed: client connection to 127.0.57.194:40207: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry. W0507 06:32:26.615881 9920 consensus_peers.cc:457] T 5ae09859764b4956baeac1f4c5241aed P 0390998780894750a0e7904a91af1410 -> Peer b46048beca56499c8ee771446de3cbd4 (127.0.57.193:46277): Couldn't send request to peer b46048beca56499c8ee771446de3cbd4. Status: Network error: Client connection negotiation failed: client connection to 127.0.57.193:46277: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry. W0507 06:32:26.834736 9918 consensus_peers.cc:457] T 5ae09859764b4956baeac1f4c5241aed P 0390998780894750a0e7904a91af1410 -> Peer 0198be74df5a45dea84591d6b6c20c66 (127.0.57.194:40207): Couldn't send request to peer 0198be74df5a45dea84591d6b6c20c66. Status: Network error: Client connection negotiation failed: client connection to 127.0.57.194:40207: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry. W0507 06:32:26.852705 9920 consensus_peers.cc:457] T 5ae09859764b4956baeac1f4c5241aed P 0390998780894750a0e7904a91af1410 -> Peer b46048beca56499c8ee771446de3cbd4 (127.0.57.193:46277): Couldn't send request to peer b46048beca56499c8ee771446de3cbd4. Status: Network error: Client connection negotiation failed: client connection to 127.0.57.193:46277: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry. W0507 06:32:27.094290 9918 consensus_peers.cc:457] T 5ae09859764b4956baeac1f4c5241aed P 0390998780894750a0e7904a91af1410 -> Peer 0198be74df5a45dea84591d6b6c20c66 (127.0.57.194:40207): Couldn't send request to peer 0198be74df5a45dea84591d6b6c20c66. Status: Network error: Client connection negotiation failed: client connection to 127.0.57.194:40207: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry. W0507 06:32:27.107110 9920 consensus_peers.cc:457] T 5ae09859764b4956baeac1f4c5241aed P 0390998780894750a0e7904a91af1410 -> Peer b46048beca56499c8ee771446de3cbd4 (127.0.57.193:46277): Couldn't send request to peer b46048beca56499c8ee771446de3cbd4. Status: Network error: Client connection negotiation failed: client connection to 127.0.57.193:46277: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry. W0507 06:32:27.318552 9918 consensus_peers.cc:457] T 5ae09859764b4956baeac1f4c5241aed P 0390998780894750a0e7904a91af1410 -> Peer 0198be74df5a45dea84591d6b6c20c66 (127.0.57.194:40207): Couldn't send request to peer 0198be74df5a45dea84591d6b6c20c66. Status: Network error: Client connection negotiation failed: client connection to 127.0.57.194:40207: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry. W0507 06:32:27.371053 9920 consensus_peers.cc:457] T 5ae09859764b4956baeac1f4c5241aed P 0390998780894750a0e7904a91af1410 -> Peer b46048beca56499c8ee771446de3cbd4 (127.0.57.193:46277): Couldn't send request to peer b46048beca56499c8ee771446de3cbd4. Status: Network error: Client connection negotiation failed: client connection to 127.0.57.193:46277: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry. W0507 06:32:27.553187 9918 consensus_peers.cc:457] T 5ae09859764b4956baeac1f4c5241aed P 0390998780894750a0e7904a91af1410 -> Peer 0198be74df5a45dea84591d6b6c20c66 (127.0.57.194:40207): Couldn't send request to peer 0198be74df5a45dea84591d6b6c20c66. Status: Network error: Client connection negotiation failed: client connection to 127.0.57.194:40207: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry. I0507 06:32:27.609534 231 external_mini_cluster.cc:933] Running /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.NonVoterReplicasInConsensusQueue.1557210500371863-231/raft_consensus-itest-cluster/ts-0/wal --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.NonVoterReplicasInConsensusQueue.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data --block_manager=log --rpc_bind_addresses=127.0.57.193:46277 --local_ip_for_outbound_sockets=127.0.57.193 --webserver_port=44431 --webserver_interface=127.0.57.193 --tserver_master_addrs=127.0.57.254:32871 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.NonVoterReplicasInConsensusQueue.1557210500371863-231/raft_consensus-itest-cluster/ts-0/logs --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.NonVoterReplicasInConsensusQueue.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_heartbeat_interval_ms=50 with env {} W0507 06:32:27.640002 9920 consensus_peers.cc:457] T 5ae09859764b4956baeac1f4c5241aed P 0390998780894750a0e7904a91af1410 -> Peer b46048beca56499c8ee771446de3cbd4 (127.0.57.193:46277): Couldn't send request to peer b46048beca56499c8ee771446de3cbd4. Status: Network error: Client connection negotiation failed: client connection to 127.0.57.193:46277: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry. W0507 06:32:27.780447 9918 consensus_peers.cc:457] T 5ae09859764b4956baeac1f4c5241aed P 0390998780894750a0e7904a91af1410 -> Peer 0198be74df5a45dea84591d6b6c20c66 (127.0.57.194:40207): Couldn't send request to peer 0198be74df5a45dea84591d6b6c20c66. Status: Network error: Client connection negotiation failed: client connection to 127.0.57.194:40207: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry. WARNING: Logging before InitGoogleLogging() is written to STDERR W0507 06:32:27.785212 10084 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0507 06:32:27.785498 10084 flags.cc:404] Enabled unsafe flag: --never_fsync=true W0507 06:32:27.787748 10084 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W0507 06:32:27.787847 10084 flags.cc:404] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.57.193 I0507 06:32:27.792060 10084 tablet_server_main.cc:78] Tablet server non-default flags: --raft_heartbeat_interval_ms=50 --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.NonVoterReplicasInConsensusQueue.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.NonVoterReplicasInConsensusQueue.1557210500371863-231/raft_consensus-itest-cluster/ts-0/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.0.57.193:46277 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.NonVoterReplicasInConsensusQueue.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data/info.pb --webserver_interface=127.0.57.193 --webserver_port=44431 --tserver_master_addrs=127.0.57.254:32871 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.0.57.193 --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.NonVoterReplicasInConsensusQueue.1557210500371863-231/raft_consensus-itest-cluster/ts-0/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.10.0-cdh6.x-SNAPSHOT revision 7d0ee26697926e87af9ef395008799577f6ab43f build type FASTDEBUG built by jenkins at 06 May 2019 22:44:21 PST on quasar-knmpfx-1.vpc.cloudera.com build id 322017 ASAN enabled I0507 06:32:27.800981 10084 system_ntp.cc:202] NTP initialized. Skew: 500ppm Current error: 148016us I0507 06:32:27.801697 10084 fs_manager.cc:263] Metadata directory not provided I0507 06:32:27.801793 10084 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0507 06:32:27.807919 10084 fs_manager.cc:399] Time spent opening directory manager: real 0.005s user 0.006s sys 0.000s I0507 06:32:27.808681 10084 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0507 06:32:27.809306 10084 file_cache.cc:466] Constructed file cache lbm with capacity 419430 I0507 06:32:27.812482 10084 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s I0507 06:32:27.812602 10084 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.NonVoterReplicasInConsensusQueue.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.NonVoterReplicasInConsensusQueue.1557210500371863-231/raft_consensus-itest-cluster/ts-0/wal uuid: "b46048beca56499c8ee771446de3cbd4" format_stamp: "Formatted at 2019-05-07 06:32:21 on dist-test-slave-dist-test-slave-rqp6" I0507 06:32:27.813694 10084 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.NonVoterReplicasInConsensusQueue.1557210500371863-231/raft_consensus-itest-cluster/ts-0/wal metadata directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.NonVoterReplicasInConsensusQueue.1557210500371863-231/raft_consensus-itest-cluster/ts-0/wal 1 data directories: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.NonVoterReplicasInConsensusQueue.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0507 06:32:27.876348 10084 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0507 06:32:27.878733 10084 ts_tablet_manager.cc:344] Loading tablet metadata (0/1 complete) W0507 06:32:27.880266 9920 consensus_peers.cc:457] T 5ae09859764b4956baeac1f4c5241aed P 0390998780894750a0e7904a91af1410 -> Peer b46048beca56499c8ee771446de3cbd4 (127.0.57.193:46277): Couldn't send request to peer b46048beca56499c8ee771446de3cbd4. Status: Network error: Client connection negotiation failed: client connection to 127.0.57.193:46277: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry. I0507 06:32:27.883195 10084 ts_tablet_manager.cc:356] Loaded tablet metadata (1 total tablets, 1 live tablets) I0507 06:32:27.883270 10084 ts_tablet_manager.cc:362] Registering tablets (0/1 complete) I0507 06:32:27.886390 10100 tablet_bootstrap.cc:439] T 5ae09859764b4956baeac1f4c5241aed P b46048beca56499c8ee771446de3cbd4: Bootstrap starting. I0507 06:32:27.896704 10084 ts_tablet_manager.cc:376] Registered 1 tablets I0507 06:32:27.904366 10100 log.cc:525] T 5ae09859764b4956baeac1f4c5241aed P b46048beca56499c8ee771446de3cbd4: Log is configured to *not* fsync() on all Append() calls I0507 06:32:27.908730 10100 tablet_bootstrap.cc:439] T 5ae09859764b4956baeac1f4c5241aed P b46048beca56499c8ee771446de3cbd4: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I0507 06:32:27.909283 10100 tablet_bootstrap.cc:439] T 5ae09859764b4956baeac1f4c5241aed P b46048beca56499c8ee771446de3cbd4: Bootstrap complete. I0507 06:32:27.910495 10100 ts_tablet_manager.cc:1053] T 5ae09859764b4956baeac1f4c5241aed P b46048beca56499c8ee771446de3cbd4: Time spent bootstrapping tablet: real 0.024s user 0.008s sys 0.010s I0507 06:32:27.926750 10100 raft_consensus.cc:340] T 5ae09859764b4956baeac1f4c5241aed P b46048beca56499c8ee771446de3cbd4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b46048beca56499c8ee771446de3cbd4" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 46277 } } peers { permanent_uuid: "0198be74df5a45dea84591d6b6c20c66" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 40207 } } peers { permanent_uuid: "0390998780894750a0e7904a91af1410" member_type: VOTER last_known_addr { host: "127.0.57.196" port: 33671 } } I0507 06:32:27.930960 10100 raft_consensus.cc:705] T 5ae09859764b4956baeac1f4c5241aed P b46048beca56499c8ee771446de3cbd4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b46048beca56499c8ee771446de3cbd4, State: Initialized, Role: FOLLOWER I0507 06:32:27.931690 10100 consensus_queue.cc:229] T 5ae09859764b4956baeac1f4c5241aed P b46048beca56499c8ee771446de3cbd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b46048beca56499c8ee771446de3cbd4" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 46277 } } peers { permanent_uuid: "0198be74df5a45dea84591d6b6c20c66" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 40207 } } peers { permanent_uuid: "0390998780894750a0e7904a91af1410" member_type: VOTER last_known_addr { host: "127.0.57.196" port: 33671 } } I0507 06:32:27.933743 10100 ts_tablet_manager.cc:1081] T 5ae09859764b4956baeac1f4c5241aed P b46048beca56499c8ee771446de3cbd4: Time spent starting tablet: real 0.023s user 0.020s sys 0.002s I0507 06:32:27.941632 10084 rpc_server.cc:205] RPC server started. Bound to: 127.0.57.193:46277 I0507 06:32:27.942891 10084 webserver.cc:308] Webserver started at http://127.0.57.193:44431/ using document root and password file I0507 06:32:27.943645 10084 server_base.cc:608] Dumped server information to /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.NonVoterReplicasInConsensusQueue.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data/info.pb I0507 06:32:27.951246 231 external_mini_cluster.cc:995] Started /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver as pid 10084 I0507 06:32:27.951850 231 external_mini_cluster.cc:933] Running /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.NonVoterReplicasInConsensusQueue.1557210500371863-231/raft_consensus-itest-cluster/ts-1/wal --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.NonVoterReplicasInConsensusQueue.1557210500371863-231/raft_consensus-itest-cluster/ts-1/data --block_manager=log --rpc_bind_addresses=127.0.57.194:40207 --local_ip_for_outbound_sockets=127.0.57.194 --webserver_port=40567 --webserver_interface=127.0.57.194 --tserver_master_addrs=127.0.57.254:32871 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.NonVoterReplicasInConsensusQueue.1557210500371863-231/raft_consensus-itest-cluster/ts-1/logs --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.NonVoterReplicasInConsensusQueue.1557210500371863-231/raft_consensus-itest-cluster/ts-1/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_heartbeat_interval_ms=50 with env {} I0507 06:32:27.961216 10207 heartbeater.cc:345] Connected to a master server at 127.0.57.254:32871 I0507 06:32:27.961513 10207 heartbeater.cc:426] Registering TS with master... I0507 06:32:27.962164 10207 heartbeater.cc:475] Master 127.0.57.254:32871 requested a full tablet report, sending... I0507 06:32:27.964308 9498 ts_manager.cc:159] Re-registered known tserver with Master: b46048beca56499c8ee771446de3cbd4 (127.0.57.193:46277) I0507 06:32:27.965579 9498 master_service.cc:273] Signed X509 certificate for tserver {username='slave'} at 127.0.57.193:43725 I0507 06:32:27.983317 10207 heartbeater.cc:467] Master 127.0.57.254:32871 was elected leader, sending a full tablet report... I0507 06:32:27.988919 10214 raft_consensus.cc:2788] T 5ae09859764b4956baeac1f4c5241aed P 0390998780894750a0e7904a91af1410 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER a6726eec155e4dc48c6ae00643563ab4 (127.0.57.195) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b46048beca56499c8ee771446de3cbd4" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 46277 } } peers { permanent_uuid: "0198be74df5a45dea84591d6b6c20c66" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 40207 } } peers { permanent_uuid: "0390998780894750a0e7904a91af1410" member_type: VOTER last_known_addr { host: "127.0.57.196" port: 33671 } } peers { permanent_uuid: "a6726eec155e4dc48c6ae00643563ab4" member_type: NON_VOTER last_known_addr { host: "127.0.57.195" port: 38581 } attrs { } } } I0507 06:32:27.989540 10163 raft_consensus.cc:2788] T 5ae09859764b4956baeac1f4c5241aed P b46048beca56499c8ee771446de3cbd4 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER a6726eec155e4dc48c6ae00643563ab4 (127.0.57.195) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b46048beca56499c8ee771446de3cbd4" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 46277 } } peers { permanent_uuid: "0198be74df5a45dea84591d6b6c20c66" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 40207 } } peers { permanent_uuid: "0390998780894750a0e7904a91af1410" member_type: VOTER last_known_addr { host: "127.0.57.196" port: 33671 } } peers { permanent_uuid: "a6726eec155e4dc48c6ae00643563ab4" member_type: NON_VOTER last_known_addr { host: "127.0.57.195" port: 38581 } attrs { } } } I0507 06:32:27.994873 9498 catalog_manager.cc:3988] T 5ae09859764b4956baeac1f4c5241aed P b46048beca56499c8ee771446de3cbd4 reported cstate change: config changed from index -1 to 2, NON_VOTER a6726eec155e4dc48c6ae00643563ab4 (127.0.57.195) added. New cstate: current_term: 1 leader_uuid: "0390998780894750a0e7904a91af1410" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b46048beca56499c8ee771446de3cbd4" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 46277 } } peers { permanent_uuid: "0198be74df5a45dea84591d6b6c20c66" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 40207 } } peers { permanent_uuid: "0390998780894750a0e7904a91af1410" member_type: VOTER last_known_addr { host: "127.0.57.196" port: 33671 } } peers { permanent_uuid: "a6726eec155e4dc48c6ae00643563ab4" member_type: NON_VOTER last_known_addr { host: "127.0.57.195" port: 38581 } attrs { } } } I0507 06:32:27.996779 10214 rpcz_store.cc:269] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:40754 (request call id 4) took 5393ms. Request Metrics: {"dns_us":90,"raft.queue_time_us":398,"raft.run_cpu_time_us":458,"raft.run_wall_time_us":484,"spinlock_wait_cycles":157824} I0507 06:32:27.999289 9865 raft_consensus.cc:2788] T 5ae09859764b4956baeac1f4c5241aed P a6726eec155e4dc48c6ae00643563ab4 [term 1 LEARNER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER a6726eec155e4dc48c6ae00643563ab4 (127.0.57.195) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b46048beca56499c8ee771446de3cbd4" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 46277 } } peers { permanent_uuid: "0198be74df5a45dea84591d6b6c20c66" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 40207 } } peers { permanent_uuid: "0390998780894750a0e7904a91af1410" member_type: VOTER last_known_addr { host: "127.0.57.196" port: 33671 } } peers { permanent_uuid: "a6726eec155e4dc48c6ae00643563ab4" member_type: NON_VOTER last_known_addr { host: "127.0.57.195" port: 38581 } attrs { } } } W0507 06:32:28.026681 9918 consensus_peers.cc:457] T 5ae09859764b4956baeac1f4c5241aed P 0390998780894750a0e7904a91af1410 -> Peer 0198be74df5a45dea84591d6b6c20c66 (127.0.57.194:40207): Couldn't send request to peer 0198be74df5a45dea84591d6b6c20c66. Status: Network error: Client connection negotiation failed: client connection to 127.0.57.194:40207: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry. WARNING: Logging before InitGoogleLogging() is written to STDERR W0507 06:32:28.149996 10211 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0507 06:32:28.150243 10211 flags.cc:404] Enabled unsafe flag: --never_fsync=true W0507 06:32:28.152544 10211 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W0507 06:32:28.152662 10211 flags.cc:404] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.57.194 I0507 06:32:28.156764 10211 tablet_server_main.cc:78] Tablet server non-default flags: --raft_heartbeat_interval_ms=50 --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.NonVoterReplicasInConsensusQueue.1557210500371863-231/raft_consensus-itest-cluster/ts-1/data --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.NonVoterReplicasInConsensusQueue.1557210500371863-231/raft_consensus-itest-cluster/ts-1/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.0.57.194:40207 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.NonVoterReplicasInConsensusQueue.1557210500371863-231/raft_consensus-itest-cluster/ts-1/data/info.pb --webserver_interface=127.0.57.194 --webserver_port=40567 --tserver_master_addrs=127.0.57.254:32871 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.0.57.194 --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.NonVoterReplicasInConsensusQueue.1557210500371863-231/raft_consensus-itest-cluster/ts-1/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.10.0-cdh6.x-SNAPSHOT revision 7d0ee26697926e87af9ef395008799577f6ab43f build type FASTDEBUG built by jenkins at 06 May 2019 22:44:21 PST on quasar-knmpfx-1.vpc.cloudera.com build id 322017 ASAN enabled I0507 06:32:28.164690 10211 system_ntp.cc:202] NTP initialized. Skew: 500ppm Current error: 148516us I0507 06:32:28.165956 10211 fs_manager.cc:263] Metadata directory not provided I0507 06:32:28.166092 10211 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0507 06:32:28.172920 10211 fs_manager.cc:399] Time spent opening directory manager: real 0.005s user 0.005s sys 0.002s I0507 06:32:28.173712 10211 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0507 06:32:28.174440 10211 file_cache.cc:466] Constructed file cache lbm with capacity 419430 I0507 06:32:28.177789 10211 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s I0507 06:32:28.177875 10211 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.NonVoterReplicasInConsensusQueue.1557210500371863-231/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.NonVoterReplicasInConsensusQueue.1557210500371863-231/raft_consensus-itest-cluster/ts-1/wal uuid: "0198be74df5a45dea84591d6b6c20c66" format_stamp: "Formatted at 2019-05-07 06:32:21 on dist-test-slave-dist-test-slave-rqp6" I0507 06:32:28.179080 10211 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.NonVoterReplicasInConsensusQueue.1557210500371863-231/raft_consensus-itest-cluster/ts-1/wal metadata directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.NonVoterReplicasInConsensusQueue.1557210500371863-231/raft_consensus-itest-cluster/ts-1/wal 1 data directories: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.NonVoterReplicasInConsensusQueue.1557210500371863-231/raft_consensus-itest-cluster/ts-1/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0507 06:32:28.238907 10211 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0507 06:32:28.241106 10211 ts_tablet_manager.cc:344] Loading tablet metadata (0/1 complete) I0507 06:32:28.245645 10211 ts_tablet_manager.cc:356] Loaded tablet metadata (1 total tablets, 1 live tablets) I0507 06:32:28.245756 10211 ts_tablet_manager.cc:362] Registering tablets (0/1 complete) I0507 06:32:28.249307 10237 tablet_bootstrap.cc:439] T 5ae09859764b4956baeac1f4c5241aed P 0198be74df5a45dea84591d6b6c20c66: Bootstrap starting. I0507 06:32:28.252614 10211 ts_tablet_manager.cc:376] Registered 1 tablets W0507 06:32:28.267580 9918 consensus_peers.cc:457] T 5ae09859764b4956baeac1f4c5241aed P 0390998780894750a0e7904a91af1410 -> Peer 0198be74df5a45dea84591d6b6c20c66 (127.0.57.194:40207): Couldn't send request to peer 0198be74df5a45dea84591d6b6c20c66. Status: Network error: Client connection negotiation failed: client connection to 127.0.57.194:40207: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry. I0507 06:32:28.271613 10237 log.cc:525] T 5ae09859764b4956baeac1f4c5241aed P 0198be74df5a45dea84591d6b6c20c66: Log is configured to *not* fsync() on all Append() calls I0507 06:32:28.276005 10237 tablet_bootstrap.cc:439] T 5ae09859764b4956baeac1f4c5241aed P 0198be74df5a45dea84591d6b6c20c66: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I0507 06:32:28.276449 10237 tablet_bootstrap.cc:439] T 5ae09859764b4956baeac1f4c5241aed P 0198be74df5a45dea84591d6b6c20c66: Bootstrap complete. I0507 06:32:28.277766 10237 ts_tablet_manager.cc:1053] T 5ae09859764b4956baeac1f4c5241aed P 0198be74df5a45dea84591d6b6c20c66: Time spent bootstrapping tablet: real 0.029s user 0.009s sys 0.012s I0507 06:32:28.289707 10237 raft_consensus.cc:340] T 5ae09859764b4956baeac1f4c5241aed P 0198be74df5a45dea84591d6b6c20c66 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b46048beca56499c8ee771446de3cbd4" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 46277 } } peers { permanent_uuid: "0198be74df5a45dea84591d6b6c20c66" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 40207 } } peers { permanent_uuid: "0390998780894750a0e7904a91af1410" member_type: VOTER last_known_addr { host: "127.0.57.196" port: 33671 } } I0507 06:32:28.292279 10237 raft_consensus.cc:705] T 5ae09859764b4956baeac1f4c5241aed P 0198be74df5a45dea84591d6b6c20c66 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0198be74df5a45dea84591d6b6c20c66, State: Initialized, Role: FOLLOWER I0507 06:32:28.293112 10237 consensus_queue.cc:229] T 5ae09859764b4956baeac1f4c5241aed P 0198be74df5a45dea84591d6b6c20c66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b46048beca56499c8ee771446de3cbd4" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 46277 } } peers { permanent_uuid: "0198be74df5a45dea84591d6b6c20c66" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 40207 } } peers { permanent_uuid: "0390998780894750a0e7904a91af1410" member_type: VOTER last_known_addr { host: "127.0.57.196" port: 33671 } } I0507 06:32:28.295974 10237 ts_tablet_manager.cc:1081] T 5ae09859764b4956baeac1f4c5241aed P 0198be74df5a45dea84591d6b6c20c66: Time spent starting tablet: real 0.018s user 0.011s sys 0.004s I0507 06:32:28.308512 10211 rpc_server.cc:205] RPC server started. Bound to: 127.0.57.194:40207 I0507 06:32:28.309787 10211 webserver.cc:308] Webserver started at http://127.0.57.194:40567/ using document root and password file I0507 06:32:28.310525 10211 server_base.cc:608] Dumped server information to /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.NonVoterReplicasInConsensusQueue.1557210500371863-231/raft_consensus-itest-cluster/ts-1/data/info.pb I0507 06:32:28.315691 231 external_mini_cluster.cc:995] Started /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver as pid 10211 I0507 06:32:28.318948 231 external_mini_cluster.cc:1172] Attempting to check leaks for /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver pid 10084 I0507 06:32:28.326352 10344 heartbeater.cc:345] Connected to a master server at 127.0.57.254:32871 I0507 06:32:28.326581 10344 heartbeater.cc:426] Registering TS with master... I0507 06:32:28.327139 10344 heartbeater.cc:475] Master 127.0.57.254:32871 requested a full tablet report, sending... I0507 06:32:28.328716 9497 ts_manager.cc:159] Re-registered known tserver with Master: 0198be74df5a45dea84591d6b6c20c66 (127.0.57.194:40207) I0507 06:32:28.329918 9497 master_service.cc:273] Signed X509 certificate for tserver {username='slave'} at 127.0.57.194:56113 I0507 06:32:28.332373 10204 generic_service.cc:171] Checking for leaks (request via RPC) I0507 06:32:28.364079 10344 heartbeater.cc:467] Master 127.0.57.254:32871 was elected leader, sending a full tablet report... I0507 06:32:28.365262 10301 raft_consensus.cc:2788] T 5ae09859764b4956baeac1f4c5241aed P 0198be74df5a45dea84591d6b6c20c66 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER a6726eec155e4dc48c6ae00643563ab4 (127.0.57.195) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b46048beca56499c8ee771446de3cbd4" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 46277 } } peers { permanent_uuid: "0198be74df5a45dea84591d6b6c20c66" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 40207 } } peers { permanent_uuid: "0390998780894750a0e7904a91af1410" member_type: VOTER last_known_addr { host: "127.0.57.196" port: 33671 } } peers { permanent_uuid: "a6726eec155e4dc48c6ae00643563ab4" member_type: NON_VOTER last_known_addr { host: "127.0.57.195" port: 38581 } attrs { } } } I0507 06:32:28.388661 231 external_mini_cluster.cc:1131] Killing /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver with pid 10084 I0507 06:32:28.399791 231 external_mini_cluster.cc:1172] Attempting to check leaks for /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver pid 10211 W0507 06:32:28.400617 9920 consensus_peers.cc:457] T 5ae09859764b4956baeac1f4c5241aed P 0390998780894750a0e7904a91af1410 -> Peer b46048beca56499c8ee771446de3cbd4 (127.0.57.193:46277): Couldn't send request to peer b46048beca56499c8ee771446de3cbd4. Status: Network error: Client connection negotiation failed: client connection to 127.0.57.193:46277: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry. I0507 06:32:28.404947 10341 generic_service.cc:171] Checking for leaks (request via RPC) I0507 06:32:28.456342 231 external_mini_cluster.cc:1131] Killing /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver with pid 10211 I0507 06:32:28.466490 231 external_mini_cluster.cc:1172] Attempting to check leaks for /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver pid 9789 I0507 06:32:28.467443 9905 generic_service.cc:171] Checking for leaks (request via RPC) W0507 06:32:28.504830 9918 consensus_peers.cc:457] T 5ae09859764b4956baeac1f4c5241aed P 0390998780894750a0e7904a91af1410 -> Peer 0198be74df5a45dea84591d6b6c20c66 (127.0.57.194:40207): Couldn't send request to peer 0198be74df5a45dea84591d6b6c20c66. Status: Network error: Client connection negotiation failed: client connection to 127.0.57.194:40207: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry. I0507 06:32:28.520030 231 external_mini_cluster.cc:1131] Killing /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver with pid 9789 I0507 06:32:28.530951 231 external_mini_cluster.cc:1172] Attempting to check leaks for /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver pid 9911 I0507 06:32:28.531940 10027 generic_service.cc:171] Checking for leaks (request via RPC) W0507 06:32:28.591531 9921 consensus_peers.cc:457] T 5ae09859764b4956baeac1f4c5241aed P 0390998780894750a0e7904a91af1410 -> Peer a6726eec155e4dc48c6ae00643563ab4 (127.0.57.195:38581): Couldn't send request to peer a6726eec155e4dc48c6ae00643563ab4. Status: Network error: Client connection negotiation failed: client connection to 127.0.57.195:38581: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry. I0507 06:32:28.592087 231 external_mini_cluster.cc:1131] Killing /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver with pid 9911 I0507 06:32:28.604271 231 external_mini_cluster.cc:1172] Attempting to check leaks for /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-master pid 9473 I0507 06:32:28.605480 9529 generic_service.cc:171] Checking for leaks (request via RPC) I0507 06:32:28.682046 231 external_mini_cluster.cc:1131] Killing /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-master with pid 9473 [ OK ] RaftConsensusNonVoterITest.NonVoterReplicasInConsensusQueue (7890 ms) [----------] 13 tests from RaftConsensusNonVoterITest (248283 ms total) [----------] 4 tests from IncompatibleReplicaReplacementSchemesITest [ RUN ] IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig/0 I0507 06:32:28.708726 231 test_util.cc:212] Using random seed: 1636067167 I0507 06:32:28.710419 231 ts_itest-base.cc:112] Starting cluster with: I0507 06:32:28.710461 231 ts_itest-base.cc:113] -------------- I0507 06:32:28.710496 231 ts_itest-base.cc:114] 1 tablet servers I0507 06:32:28.710525 231 ts_itest-base.cc:115] 1 replicas per TS I0507 06:32:28.710562 231 ts_itest-base.cc:116] -------------- I0507 06:32:28.711946 231 external_mini_cluster.cc:933] Running /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-master /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-master --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_0.1557210500371863-231/raft_consensus-itest-cluster/master-0/wal --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_0.1557210500371863-231/raft_consensus-itest-cluster/master-0/data --block_manager=log --webserver_interface=localhost --ipki_ca_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.0.57.254:39193 --webserver_interface=127.0.57.254 --webserver_port=0 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_0.1557210500371863-231/raft_consensus-itest-cluster/master-0/logs --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_0.1557210500371863-231/raft_consensus-itest-cluster/master-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=false --rpc_reuseport=true with env {} WARNING: Logging before InitGoogleLogging() is written to STDERR W0507 06:32:28.925122 10362 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0507 06:32:28.925344 10362 flags.cc:404] Enabled unsafe flag: --never_fsync=true W0507 06:32:28.927810 10362 flags.cc:404] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false W0507 06:32:28.927891 10362 flags.cc:404] Enabled experimental flag: --ipki_ca_key_size=1024 W0507 06:32:28.927924 10362 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W0507 06:32:28.927960 10362 flags.cc:404] Enabled experimental flag: --tsk_num_rsa_bits=512 W0507 06:32:28.927990 10362 flags.cc:404] Enabled experimental flag: --rpc_reuseport=true I0507 06:32:28.932754 10362 master_main.cc:73] Master server non-default flags: --raft_prepare_replacement_before_eviction=false --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_0.1557210500371863-231/raft_consensus-itest-cluster/master-0/data --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_0.1557210500371863-231/raft_consensus-itest-cluster/master-0/wal --ipki_ca_key_size=1024 --ipki_server_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.0.57.254:39193 --rpc_reuseport=true --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_0.1557210500371863-231/raft_consensus-itest-cluster/master-0/data/info.pb --webserver_interface=127.0.57.254 --webserver_port=0 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_0.1557210500371863-231/raft_consensus-itest-cluster/master-0/logs --logbuflevel=-1 --logtostderr=true Master server version: kudu 1.10.0-cdh6.x-SNAPSHOT revision 7d0ee26697926e87af9ef395008799577f6ab43f build type FASTDEBUG built by jenkins at 06 May 2019 22:44:21 PST on quasar-knmpfx-1.vpc.cloudera.com build id 322017 ASAN enabled I0507 06:32:28.939656 10362 system_ntp.cc:202] NTP initialized. Skew: 500ppm Current error: 148516us I0507 06:32:28.941018 10362 fs_manager.cc:263] Metadata directory not provided I0507 06:32:28.941087 10362 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0507 06:32:28.941377 10362 server_base.cc:433] Could not load existing FS layout: Not found: could not find a healthy instance file I0507 06:32:28.941447 10362 server_base.cc:434] Attempting to create new FS layout instead I0507 06:32:28.943892 10362 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_0.1557210500371863-231/raft_consensus-itest-cluster/master-0/data/instance: uuid: "a147d0d4602245aaa8b2898f9bdd3f92" format_stamp: "Formatted at 2019-05-07 06:32:28 on dist-test-slave-dist-test-slave-rqp6" I0507 06:32:28.944435 10362 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_0.1557210500371863-231/raft_consensus-itest-cluster/master-0/wal/instance: uuid: "a147d0d4602245aaa8b2898f9bdd3f92" format_stamp: "Formatted at 2019-05-07 06:32:28 on dist-test-slave-dist-test-slave-rqp6" I0507 06:32:28.950887 10362 fs_manager.cc:503] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s I0507 06:32:28.951930 10362 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0507 06:32:28.952564 10362 file_cache.cc:466] Constructed file cache lbm with capacity 419430 I0507 06:32:28.955629 10362 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.000s sys 0.000s I0507 06:32:28.955727 10362 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_0.1557210500371863-231/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_0.1557210500371863-231/raft_consensus-itest-cluster/master-0/wal uuid: "a147d0d4602245aaa8b2898f9bdd3f92" format_stamp: "Formatted at 2019-05-07 06:32:28 on dist-test-slave-dist-test-slave-rqp6" I0507 06:32:28.955869 10362 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_0.1557210500371863-231/raft_consensus-itest-cluster/master-0/wal metadata directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_0.1557210500371863-231/raft_consensus-itest-cluster/master-0/wal 1 data directories: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_0.1557210500371863-231/raft_consensus-itest-cluster/master-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0507 06:32:28.981945 10362 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0507 06:32:28.995993 10362 rpc_server.cc:205] RPC server started. Bound to: 127.0.57.254:39193 I0507 06:32:28.997087 10362 webserver.cc:308] Webserver started at http://127.0.57.254:42617/ using document root and password file I0507 06:32:28.997798 10362 server_base.cc:608] Dumped server information to /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_0.1557210500371863-231/raft_consensus-itest-cluster/master-0/data/info.pb I0507 06:32:29.001381 10421 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0507 06:32:29.001543 231 external_mini_cluster.cc:995] Started /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-master as pid 10362 I0507 06:32:29.002221 231 external_mini_cluster.cc:933] Running /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_0.1557210500371863-231/raft_consensus-itest-cluster/ts-0/wal --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_0.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data --block_manager=log --rpc_bind_addresses=127.0.57.193:0 --local_ip_for_outbound_sockets=127.0.57.193 --webserver_interface=127.0.57.193 --webserver_port=0 --tserver_master_addrs=127.0.57.254:39193 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_0.1557210500371863-231/raft_consensus-itest-cluster/ts-0/logs --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_0.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --heartbeat_incompatible_replica_management_is_fatal=false --heartbeat_interval_ms=500 --raft_prepare_replacement_before_eviction=false with env {} I0507 06:32:29.012053 10421 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P a147d0d4602245aaa8b2898f9bdd3f92: Bootstrap starting. I0507 06:32:29.018906 10421 tablet_bootstrap.cc:590] T 00000000000000000000000000000000 P a147d0d4602245aaa8b2898f9bdd3f92: No blocks or log segments found. Creating new log. I0507 06:32:29.019847 10421 log.cc:525] T 00000000000000000000000000000000 P a147d0d4602245aaa8b2898f9bdd3f92: Log is configured to *not* fsync() on all Append() calls I0507 06:32:29.034333 10421 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P a147d0d4602245aaa8b2898f9bdd3f92: No bootstrap required, opened a new log I0507 06:32:29.041743 10421 raft_consensus.cc:340] T 00000000000000000000000000000000 P a147d0d4602245aaa8b2898f9bdd3f92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a147d0d4602245aaa8b2898f9bdd3f92" member_type: VOTER } I0507 06:32:29.042026 10421 raft_consensus.cc:366] T 00000000000000000000000000000000 P a147d0d4602245aaa8b2898f9bdd3f92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0507 06:32:29.042083 10421 raft_consensus.cc:705] T 00000000000000000000000000000000 P a147d0d4602245aaa8b2898f9bdd3f92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a147d0d4602245aaa8b2898f9bdd3f92, State: Initialized, Role: FOLLOWER I0507 06:32:29.042685 10421 consensus_queue.cc:229] T 00000000000000000000000000000000 P a147d0d4602245aaa8b2898f9bdd3f92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a147d0d4602245aaa8b2898f9bdd3f92" member_type: VOTER } I0507 06:32:29.042891 10421 raft_consensus.cc:380] T 00000000000000000000000000000000 P a147d0d4602245aaa8b2898f9bdd3f92 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0507 06:32:29.042973 10421 raft_consensus.cc:472] T 00000000000000000000000000000000 P a147d0d4602245aaa8b2898f9bdd3f92 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0507 06:32:29.043072 10421 raft_consensus.cc:2886] T 00000000000000000000000000000000 P a147d0d4602245aaa8b2898f9bdd3f92 [term 0 FOLLOWER]: Advancing to term 1 I0507 06:32:29.044966 10421 raft_consensus.cc:494] T 00000000000000000000000000000000 P a147d0d4602245aaa8b2898f9bdd3f92 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a147d0d4602245aaa8b2898f9bdd3f92" member_type: VOTER } I0507 06:32:29.045439 10421 leader_election.cc:310] T 00000000000000000000000000000000 P a147d0d4602245aaa8b2898f9bdd3f92 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a147d0d4602245aaa8b2898f9bdd3f92; no voters: I0507 06:32:29.048902 10421 leader_election.cc:296] T 00000000000000000000000000000000 P a147d0d4602245aaa8b2898f9bdd3f92 [CANDIDATE]: Term 1 election: Requested vote from peers I0507 06:32:29.049258 10424 raft_consensus.cc:2640] T 00000000000000000000000000000000 P a147d0d4602245aaa8b2898f9bdd3f92 [term 1 FOLLOWER]: Leader election won for term 1 I0507 06:32:29.050094 10424 raft_consensus.cc:667] T 00000000000000000000000000000000 P a147d0d4602245aaa8b2898f9bdd3f92 [term 1 LEADER]: Becoming Leader. State: Replica: a147d0d4602245aaa8b2898f9bdd3f92, State: Running, Role: LEADER I0507 06:32:29.050523 10424 consensus_queue.cc:206] T 00000000000000000000000000000000 P a147d0d4602245aaa8b2898f9bdd3f92 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a147d0d4602245aaa8b2898f9bdd3f92" member_type: VOTER } I0507 06:32:29.053395 10426 sys_catalog.cc:337] T 00000000000000000000000000000000 P a147d0d4602245aaa8b2898f9bdd3f92 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a147d0d4602245aaa8b2898f9bdd3f92. Latest consensus state: current_term: 1 leader_uuid: "a147d0d4602245aaa8b2898f9bdd3f92" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a147d0d4602245aaa8b2898f9bdd3f92" member_type: VOTER } } I0507 06:32:29.053678 10426 sys_catalog.cc:340] T 00000000000000000000000000000000 P a147d0d4602245aaa8b2898f9bdd3f92 [sys.catalog]: This master's current role is: LEADER I0507 06:32:29.053568 10421 sys_catalog.cc:337] T 00000000000000000000000000000000 P a147d0d4602245aaa8b2898f9bdd3f92 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 1 leader_uuid: "a147d0d4602245aaa8b2898f9bdd3f92" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a147d0d4602245aaa8b2898f9bdd3f92" member_type: VOTER } } I0507 06:32:29.054209 10421 sys_catalog.cc:340] T 00000000000000000000000000000000 P a147d0d4602245aaa8b2898f9bdd3f92 [sys.catalog]: This master's current role is: LEADER I0507 06:32:29.054383 10425 sys_catalog.cc:337] T 00000000000000000000000000000000 P a147d0d4602245aaa8b2898f9bdd3f92 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a147d0d4602245aaa8b2898f9bdd3f92" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a147d0d4602245aaa8b2898f9bdd3f92" member_type: VOTER } } I0507 06:32:29.056265 10425 sys_catalog.cc:340] T 00000000000000000000000000000000 P a147d0d4602245aaa8b2898f9bdd3f92 [sys.catalog]: This master's current role is: LEADER I0507 06:32:29.056866 10421 sys_catalog.cc:424] T 00000000000000000000000000000000 P a147d0d4602245aaa8b2898f9bdd3f92 [sys.catalog]: configured and running, proceeding with master startup. I0507 06:32:29.057199 10429 catalog_manager.cc:1055] Loading table and tablet metadata into memory... I0507 06:32:29.061811 10429 catalog_manager.cc:1066] Initializing Kudu internal certificate authority... I0507 06:32:29.114477 10429 catalog_manager.cc:950] Generated new certificate authority record I0507 06:32:29.115921 10429 catalog_manager.cc:1075] Loading token signing keys... I0507 06:32:29.124143 10429 catalog_manager.cc:4292] T 00000000000000000000000000000000 P a147d0d4602245aaa8b2898f9bdd3f92: Generated new TSK 0 WARNING: Logging before InitGoogleLogging() is written to STDERR W0507 06:32:29.174798 10422 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0507 06:32:29.175030 10422 flags.cc:404] Enabled unsafe flag: --never_fsync=true W0507 06:32:29.177145 10422 flags.cc:404] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false W0507 06:32:29.177209 10422 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W0507 06:32:29.177315 10422 flags.cc:404] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.57.193 I0507 06:32:29.192960 10422 tablet_server_main.cc:78] Tablet server non-default flags: --raft_prepare_replacement_before_eviction=false --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_0.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_0.1557210500371863-231/raft_consensus-itest-cluster/ts-0/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.0.57.193:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_0.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data/info.pb --webserver_interface=127.0.57.193 --webserver_port=0 --heartbeat_incompatible_replica_management_is_fatal=false --heartbeat_interval_ms=500 --tserver_master_addrs=127.0.57.254:39193 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.0.57.193 --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_0.1557210500371863-231/raft_consensus-itest-cluster/ts-0/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.10.0-cdh6.x-SNAPSHOT revision 7d0ee26697926e87af9ef395008799577f6ab43f build type FASTDEBUG built by jenkins at 06 May 2019 22:44:21 PST on quasar-knmpfx-1.vpc.cloudera.com build id 322017 ASAN enabled I0507 06:32:29.200564 10422 system_ntp.cc:202] NTP initialized. Skew: 500ppm Current error: 149016us I0507 06:32:29.201222 10422 fs_manager.cc:263] Metadata directory not provided I0507 06:32:29.201293 10422 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0507 06:32:29.201557 10422 server_base.cc:433] Could not load existing FS layout: Not found: could not find a healthy instance file I0507 06:32:29.201625 10422 server_base.cc:434] Attempting to create new FS layout instead I0507 06:32:29.203964 10422 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_0.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data/instance: uuid: "cff628f906034bfbabca7dc6fad72eda" format_stamp: "Formatted at 2019-05-07 06:32:29 on dist-test-slave-dist-test-slave-rqp6" I0507 06:32:29.204504 10422 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_0.1557210500371863-231/raft_consensus-itest-cluster/ts-0/wal/instance: uuid: "cff628f906034bfbabca7dc6fad72eda" format_stamp: "Formatted at 2019-05-07 06:32:29 on dist-test-slave-dist-test-slave-rqp6" I0507 06:32:29.215302 10422 fs_manager.cc:503] Time spent creating directory manager: real 0.011s user 0.004s sys 0.004s I0507 06:32:29.216459 10422 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0507 06:32:29.217097 10422 file_cache.cc:466] Constructed file cache lbm with capacity 419430 I0507 06:32:29.227051 10422 fs_manager.cc:419] Time spent opening block manager: real 0.009s user 0.000s sys 0.007s I0507 06:32:29.227159 10422 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_0.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_0.1557210500371863-231/raft_consensus-itest-cluster/ts-0/wal uuid: "cff628f906034bfbabca7dc6fad72eda" format_stamp: "Formatted at 2019-05-07 06:32:29 on dist-test-slave-dist-test-slave-rqp6" I0507 06:32:29.227356 10422 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_0.1557210500371863-231/raft_consensus-itest-cluster/ts-0/wal metadata directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_0.1557210500371863-231/raft_consensus-itest-cluster/ts-0/wal 1 data directories: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_0.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0507 06:32:29.257731 10422 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0507 06:32:29.259392 10422 ts_tablet_manager.cc:356] Loaded tablet metadata (0 total tablets, 0 live tablets) I0507 06:32:29.259480 10422 ts_tablet_manager.cc:376] Registered 0 tablets I0507 06:32:29.374323 10422 rpc_server.cc:205] RPC server started. Bound to: 127.0.57.193:45363 I0507 06:32:29.375728 10422 webserver.cc:308] Webserver started at http://127.0.57.193:44209/ using document root and password file I0507 06:32:29.376564 10422 server_base.cc:608] Dumped server information to /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_0.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data/info.pb I0507 06:32:29.382153 231 external_mini_cluster.cc:995] Started /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver as pid 10422 I0507 06:32:29.393386 10551 heartbeater.cc:345] Connected to a master server at 127.0.57.254:39193 I0507 06:32:29.393596 10551 heartbeater.cc:426] Registering TS with master... I0507 06:32:29.394021 10551 heartbeater.cc:475] Master 127.0.57.254:39193 requested a full tablet report, sending... I0507 06:32:29.395782 10388 ts_manager.cc:159] Registered new tserver with Master: cff628f906034bfbabca7dc6fad72eda (127.0.57.193:45363) I0507 06:32:29.397496 10388 master_service.cc:273] Signed X509 certificate for tserver {username='slave'} at 127.0.57.193:57627 I0507 06:32:29.397627 231 external_mini_cluster.cc:623] 1 TS(s) registered with all masters I0507 06:32:29.408864 10388 catalog_manager.cc:1415] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41376: name: "TestTable" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 1 split_rows_range_bounds { } partition_schema { range_schema { columns { name: "key" } } } I0507 06:32:29.433061 10488 tablet_service.cc:1016] Processing CreateTablet for tablet 1cd3f33e70bc4d9897d06e1c7ae38792 (table=TestTable [id=975742a8da5d497eb40787c75ea6d8d3]), partition=RANGE (key) PARTITION UNBOUNDED I0507 06:32:29.434099 10488 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet 1cd3f33e70bc4d9897d06e1c7ae38792. 1 dirs total, 0 dirs full, 0 dirs failed I0507 06:32:29.445010 10564 tablet_bootstrap.cc:439] T 1cd3f33e70bc4d9897d06e1c7ae38792 P cff628f906034bfbabca7dc6fad72eda: Bootstrap starting. I0507 06:32:29.447490 10564 tablet_bootstrap.cc:590] T 1cd3f33e70bc4d9897d06e1c7ae38792 P cff628f906034bfbabca7dc6fad72eda: No blocks or log segments found. Creating new log. I0507 06:32:29.448475 10564 log.cc:525] T 1cd3f33e70bc4d9897d06e1c7ae38792 P cff628f906034bfbabca7dc6fad72eda: Log is configured to *not* fsync() on all Append() calls I0507 06:32:29.450310 10564 tablet_bootstrap.cc:439] T 1cd3f33e70bc4d9897d06e1c7ae38792 P cff628f906034bfbabca7dc6fad72eda: No bootstrap required, opened a new log I0507 06:32:29.450595 10564 ts_tablet_manager.cc:1053] T 1cd3f33e70bc4d9897d06e1c7ae38792 P cff628f906034bfbabca7dc6fad72eda: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s I0507 06:32:29.456602 10564 raft_consensus.cc:340] T 1cd3f33e70bc4d9897d06e1c7ae38792 P cff628f906034bfbabca7dc6fad72eda [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cff628f906034bfbabca7dc6fad72eda" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 45363 } } I0507 06:32:29.456939 10564 raft_consensus.cc:366] T 1cd3f33e70bc4d9897d06e1c7ae38792 P cff628f906034bfbabca7dc6fad72eda [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0507 06:32:29.457007 10564 raft_consensus.cc:705] T 1cd3f33e70bc4d9897d06e1c7ae38792 P cff628f906034bfbabca7dc6fad72eda [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cff628f906034bfbabca7dc6fad72eda, State: Initialized, Role: FOLLOWER I0507 06:32:29.457484 10564 consensus_queue.cc:229] T 1cd3f33e70bc4d9897d06e1c7ae38792 P cff628f906034bfbabca7dc6fad72eda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cff628f906034bfbabca7dc6fad72eda" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 45363 } } I0507 06:32:29.457655 10564 raft_consensus.cc:380] T 1cd3f33e70bc4d9897d06e1c7ae38792 P cff628f906034bfbabca7dc6fad72eda [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0507 06:32:29.457731 10564 raft_consensus.cc:472] T 1cd3f33e70bc4d9897d06e1c7ae38792 P cff628f906034bfbabca7dc6fad72eda [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0507 06:32:29.457828 10564 raft_consensus.cc:2886] T 1cd3f33e70bc4d9897d06e1c7ae38792 P cff628f906034bfbabca7dc6fad72eda [term 0 FOLLOWER]: Advancing to term 1 I0507 06:32:29.459496 10564 raft_consensus.cc:494] T 1cd3f33e70bc4d9897d06e1c7ae38792 P cff628f906034bfbabca7dc6fad72eda [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cff628f906034bfbabca7dc6fad72eda" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 45363 } } I0507 06:32:29.459923 10564 leader_election.cc:310] T 1cd3f33e70bc4d9897d06e1c7ae38792 P cff628f906034bfbabca7dc6fad72eda [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cff628f906034bfbabca7dc6fad72eda; no voters: I0507 06:32:29.460616 10566 raft_consensus.cc:2640] T 1cd3f33e70bc4d9897d06e1c7ae38792 P cff628f906034bfbabca7dc6fad72eda [term 1 FOLLOWER]: Leader election won for term 1 I0507 06:32:29.460958 10564 leader_election.cc:296] T 1cd3f33e70bc4d9897d06e1c7ae38792 P cff628f906034bfbabca7dc6fad72eda [CANDIDATE]: Term 1 election: Requested vote from peers I0507 06:32:29.461737 10551 heartbeater.cc:467] Master 127.0.57.254:39193 was elected leader, sending a full tablet report... I0507 06:32:29.461740 10566 raft_consensus.cc:667] T 1cd3f33e70bc4d9897d06e1c7ae38792 P cff628f906034bfbabca7dc6fad72eda [term 1 LEADER]: Becoming Leader. State: Replica: cff628f906034bfbabca7dc6fad72eda, State: Running, Role: LEADER I0507 06:32:29.462076 10566 consensus_queue.cc:206] T 1cd3f33e70bc4d9897d06e1c7ae38792 P cff628f906034bfbabca7dc6fad72eda [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cff628f906034bfbabca7dc6fad72eda" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 45363 } } I0507 06:32:29.463806 10564 ts_tablet_manager.cc:1081] T 1cd3f33e70bc4d9897d06e1c7ae38792 P cff628f906034bfbabca7dc6fad72eda: Time spent starting tablet: real 0.013s user 0.008s sys 0.003s I0507 06:32:29.467998 10388 catalog_manager.cc:3988] T 1cd3f33e70bc4d9897d06e1c7ae38792 P cff628f906034bfbabca7dc6fad72eda reported cstate change: term changed from 0 to 1, leader changed from to cff628f906034bfbabca7dc6fad72eda (127.0.57.193). New cstate: current_term: 1 leader_uuid: "cff628f906034bfbabca7dc6fad72eda" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cff628f906034bfbabca7dc6fad72eda" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 45363 } } } I0507 06:32:29.490347 231 external_mini_cluster.cc:623] 1 TS(s) registered with all masters I0507 06:32:29.491726 231 ts_itest-base.cc:232] Waiting for 1 tablets on tserver cff628f906034bfbabca7dc6fad72eda to finish bootstrapping I0507 06:32:29.499469 231 external_mini_cluster.cc:1172] Attempting to check leaks for /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver pid 10422 I0507 06:32:29.500300 10548 generic_service.cc:171] Checking for leaks (request via RPC) I0507 06:32:29.553182 231 external_mini_cluster.cc:1131] Killing /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver with pid 10422 I0507 06:32:29.565174 231 external_mini_cluster.cc:1172] Attempting to check leaks for /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-master pid 10362 I0507 06:32:29.566321 10418 generic_service.cc:171] Checking for leaks (request via RPC) I0507 06:32:29.650198 231 external_mini_cluster.cc:1131] Killing /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-master with pid 10362 I0507 06:32:29.666956 231 external_mini_cluster.cc:933] Running /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-master /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-master --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_0.1557210500371863-231/raft_consensus-itest-cluster/master-0/wal --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_0.1557210500371863-231/raft_consensus-itest-cluster/master-0/data --block_manager=log --webserver_interface=localhost --ipki_ca_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.0.57.254:39193 --webserver_interface=127.0.57.254 --webserver_port=42617 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_0.1557210500371863-231/raft_consensus-itest-cluster/master-0/logs --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_0.1557210500371863-231/raft_consensus-itest-cluster/master-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=false --rpc_reuseport=true with env {} WARNING: Logging before InitGoogleLogging() is written to STDERR W0507 06:32:29.880892 10573 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0507 06:32:29.881156 10573 flags.cc:404] Enabled unsafe flag: --never_fsync=true W0507 06:32:29.883652 10573 flags.cc:404] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false W0507 06:32:29.883718 10573 flags.cc:404] Enabled experimental flag: --ipki_ca_key_size=1024 W0507 06:32:29.883752 10573 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W0507 06:32:29.883785 10573 flags.cc:404] Enabled experimental flag: --tsk_num_rsa_bits=512 W0507 06:32:29.883823 10573 flags.cc:404] Enabled experimental flag: --rpc_reuseport=true I0507 06:32:29.888706 10573 master_main.cc:73] Master server non-default flags: --raft_prepare_replacement_before_eviction=false --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_0.1557210500371863-231/raft_consensus-itest-cluster/master-0/data --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_0.1557210500371863-231/raft_consensus-itest-cluster/master-0/wal --ipki_ca_key_size=1024 --ipki_server_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.0.57.254:39193 --rpc_reuseport=true --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_0.1557210500371863-231/raft_consensus-itest-cluster/master-0/data/info.pb --webserver_interface=127.0.57.254 --webserver_port=42617 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_0.1557210500371863-231/raft_consensus-itest-cluster/master-0/logs --logbuflevel=-1 --logtostderr=true Master server version: kudu 1.10.0-cdh6.x-SNAPSHOT revision 7d0ee26697926e87af9ef395008799577f6ab43f build type FASTDEBUG built by jenkins at 06 May 2019 22:44:21 PST on quasar-knmpfx-1.vpc.cloudera.com build id 322017 ASAN enabled I0507 06:32:29.896481 10573 system_ntp.cc:202] NTP initialized. Skew: 500ppm Current error: 149016us I0507 06:32:29.897924 10573 fs_manager.cc:263] Metadata directory not provided I0507 06:32:29.898011 10573 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0507 06:32:29.904405 10573 fs_manager.cc:399] Time spent opening directory manager: real 0.004s user 0.001s sys 0.005s I0507 06:32:29.905115 10573 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0507 06:32:29.905699 10573 file_cache.cc:466] Constructed file cache lbm with capacity 419430 I0507 06:32:29.908836 10573 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.000s sys 0.000s I0507 06:32:29.908928 10573 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_0.1557210500371863-231/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_0.1557210500371863-231/raft_consensus-itest-cluster/master-0/wal uuid: "a147d0d4602245aaa8b2898f9bdd3f92" format_stamp: "Formatted at 2019-05-07 06:32:28 on dist-test-slave-dist-test-slave-rqp6" I0507 06:32:29.909962 10573 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_0.1557210500371863-231/raft_consensus-itest-cluster/master-0/wal metadata directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_0.1557210500371863-231/raft_consensus-itest-cluster/master-0/wal 1 data directories: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_0.1557210500371863-231/raft_consensus-itest-cluster/master-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0507 06:32:29.956367 10573 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0507 06:32:29.972273 10573 rpc_server.cc:205] RPC server started. Bound to: 127.0.57.254:39193 I0507 06:32:29.973672 10573 webserver.cc:308] Webserver started at http://127.0.57.254:42617/ using document root and password file I0507 06:32:29.974642 10573 server_base.cc:608] Dumped server information to /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_0.1557210500371863-231/raft_consensus-itest-cluster/master-0/data/info.pb I0507 06:32:29.980299 231 external_mini_cluster.cc:995] Started /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-master as pid 10573 I0507 06:32:29.981110 231 external_mini_cluster.cc:933] Running /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_0.1557210500371863-231/raft_consensus-itest-cluster/ts-0/wal --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_0.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data --block_manager=log --rpc_bind_addresses=127.0.57.193:45363 --local_ip_for_outbound_sockets=127.0.57.193 --webserver_port=44209 --webserver_interface=127.0.57.193 --tserver_master_addrs=127.0.57.254:39193 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_0.1557210500371863-231/raft_consensus-itest-cluster/ts-0/logs --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_0.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --heartbeat_incompatible_replica_management_is_fatal=false --heartbeat_interval_ms=500 --raft_prepare_replacement_before_eviction=false --raft_prepare_replacement_before_eviction=true with env {} I0507 06:32:29.984421 10632 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P a147d0d4602245aaa8b2898f9bdd3f92: Bootstrap starting. I0507 06:32:30.001469 10632 log.cc:525] T 00000000000000000000000000000000 P a147d0d4602245aaa8b2898f9bdd3f92: Log is configured to *not* fsync() on all Append() calls I0507 06:32:30.014919 10632 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P a147d0d4602245aaa8b2898f9bdd3f92: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=4 ignored=0} mutations{seen=3 ignored=0} orphaned_commits=0. Pending: 0 replicates I0507 06:32:30.015651 10632 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P a147d0d4602245aaa8b2898f9bdd3f92: Bootstrap complete. I0507 06:32:30.026932 10632 raft_consensus.cc:340] T 00000000000000000000000000000000 P a147d0d4602245aaa8b2898f9bdd3f92 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a147d0d4602245aaa8b2898f9bdd3f92" member_type: VOTER } I0507 06:32:30.027773 10632 raft_consensus.cc:705] T 00000000000000000000000000000000 P a147d0d4602245aaa8b2898f9bdd3f92 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a147d0d4602245aaa8b2898f9bdd3f92, State: Initialized, Role: FOLLOWER I0507 06:32:30.028396 10632 consensus_queue.cc:229] T 00000000000000000000000000000000 P a147d0d4602245aaa8b2898f9bdd3f92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a147d0d4602245aaa8b2898f9bdd3f92" member_type: VOTER } I0507 06:32:30.028609 10632 raft_consensus.cc:380] T 00000000000000000000000000000000 P a147d0d4602245aaa8b2898f9bdd3f92 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0507 06:32:30.028710 10632 raft_consensus.cc:472] T 00000000000000000000000000000000 P a147d0d4602245aaa8b2898f9bdd3f92 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0507 06:32:30.028829 10632 raft_consensus.cc:2886] T 00000000000000000000000000000000 P a147d0d4602245aaa8b2898f9bdd3f92 [term 1 FOLLOWER]: Advancing to term 2 I0507 06:32:30.033188 10632 raft_consensus.cc:494] T 00000000000000000000000000000000 P a147d0d4602245aaa8b2898f9bdd3f92 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a147d0d4602245aaa8b2898f9bdd3f92" member_type: VOTER } I0507 06:32:30.033710 10632 leader_election.cc:310] T 00000000000000000000000000000000 P a147d0d4602245aaa8b2898f9bdd3f92 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a147d0d4602245aaa8b2898f9bdd3f92; no voters: I0507 06:32:30.036754 10632 leader_election.cc:296] T 00000000000000000000000000000000 P a147d0d4602245aaa8b2898f9bdd3f92 [CANDIDATE]: Term 2 election: Requested vote from peers I0507 06:32:30.036959 10635 raft_consensus.cc:2640] T 00000000000000000000000000000000 P a147d0d4602245aaa8b2898f9bdd3f92 [term 2 FOLLOWER]: Leader election won for term 2 I0507 06:32:30.040743 10635 raft_consensus.cc:667] T 00000000000000000000000000000000 P a147d0d4602245aaa8b2898f9bdd3f92 [term 2 LEADER]: Becoming Leader. State: Replica: a147d0d4602245aaa8b2898f9bdd3f92, State: Running, Role: LEADER I0507 06:32:30.041362 10635 consensus_queue.cc:206] T 00000000000000000000000000000000 P a147d0d4602245aaa8b2898f9bdd3f92 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a147d0d4602245aaa8b2898f9bdd3f92" member_type: VOTER } I0507 06:32:30.045706 10637 sys_catalog.cc:337] T 00000000000000000000000000000000 P a147d0d4602245aaa8b2898f9bdd3f92 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "a147d0d4602245aaa8b2898f9bdd3f92" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a147d0d4602245aaa8b2898f9bdd3f92" member_type: VOTER } } I0507 06:32:30.046289 10637 sys_catalog.cc:340] T 00000000000000000000000000000000 P a147d0d4602245aaa8b2898f9bdd3f92 [sys.catalog]: This master's current role is: LEADER I0507 06:32:30.046355 10632 sys_catalog.cc:337] T 00000000000000000000000000000000 P a147d0d4602245aaa8b2898f9bdd3f92 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 2 leader_uuid: "a147d0d4602245aaa8b2898f9bdd3f92" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a147d0d4602245aaa8b2898f9bdd3f92" member_type: VOTER } } I0507 06:32:30.046730 10632 sys_catalog.cc:340] T 00000000000000000000000000000000 P a147d0d4602245aaa8b2898f9bdd3f92 [sys.catalog]: This master's current role is: LEADER I0507 06:32:30.045609 10636 sys_catalog.cc:337] T 00000000000000000000000000000000 P a147d0d4602245aaa8b2898f9bdd3f92 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a147d0d4602245aaa8b2898f9bdd3f92. Latest consensus state: current_term: 2 leader_uuid: "a147d0d4602245aaa8b2898f9bdd3f92" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a147d0d4602245aaa8b2898f9bdd3f92" member_type: VOTER } } I0507 06:32:30.048419 10636 sys_catalog.cc:340] T 00000000000000000000000000000000 P a147d0d4602245aaa8b2898f9bdd3f92 [sys.catalog]: This master's current role is: LEADER I0507 06:32:30.049024 10632 sys_catalog.cc:424] T 00000000000000000000000000000000 P a147d0d4602245aaa8b2898f9bdd3f92 [sys.catalog]: configured and running, proceeding with master startup. I0507 06:32:30.049177 10640 catalog_manager.cc:1055] Loading table and tablet metadata into memory... I0507 06:32:30.055729 10640 catalog_manager.cc:356] Loaded metadata for table TestTable [id=975742a8da5d497eb40787c75ea6d8d3] I0507 06:32:30.058080 10640 catalog_manager.cc:410] Loaded metadata for tablet 1cd3f33e70bc4d9897d06e1c7ae38792 (table TestTable [id=975742a8da5d497eb40787c75ea6d8d3]) I0507 06:32:30.058353 10640 catalog_manager.cc:1066] Initializing Kudu internal certificate authority... I0507 06:32:30.062710 10640 catalog_manager.cc:1075] Loading token signing keys... I0507 06:32:30.064177 10640 catalog_manager.cc:4303] T 00000000000000000000000000000000 P a147d0d4602245aaa8b2898f9bdd3f92: Loaded TSK: 0 WARNING: Logging before InitGoogleLogging() is written to STDERR W0507 06:32:30.179531 10633 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0507 06:32:30.179893 10633 flags.cc:404] Enabled unsafe flag: --never_fsync=true W0507 06:32:30.183174 10633 flags.cc:404] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true W0507 06:32:30.183284 10633 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W0507 06:32:30.183470 10633 flags.cc:404] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.57.193 I0507 06:32:30.189460 10633 tablet_server_main.cc:78] Tablet server non-default flags: --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_0.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_0.1557210500371863-231/raft_consensus-itest-cluster/ts-0/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.0.57.193:45363 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_0.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data/info.pb --webserver_interface=127.0.57.193 --webserver_port=44209 --heartbeat_incompatible_replica_management_is_fatal=false --heartbeat_interval_ms=500 --tserver_master_addrs=127.0.57.254:39193 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.0.57.193 --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_0.1557210500371863-231/raft_consensus-itest-cluster/ts-0/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.10.0-cdh6.x-SNAPSHOT revision 7d0ee26697926e87af9ef395008799577f6ab43f build type FASTDEBUG built by jenkins at 06 May 2019 22:44:21 PST on quasar-knmpfx-1.vpc.cloudera.com build id 322017 ASAN enabled I0507 06:32:30.199916 10633 system_ntp.cc:202] NTP initialized. Skew: 500ppm Current error: 149516us I0507 06:32:30.200745 10633 fs_manager.cc:263] Metadata directory not provided I0507 06:32:30.200825 10633 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0507 06:32:30.211269 10633 fs_manager.cc:399] Time spent opening directory manager: real 0.009s user 0.005s sys 0.004s I0507 06:32:30.212246 10633 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0507 06:32:30.213104 10633 file_cache.cc:466] Constructed file cache lbm with capacity 419430 I0507 06:32:30.222528 10633 fs_manager.cc:419] Time spent opening block manager: real 0.008s user 0.000s sys 0.007s I0507 06:32:30.222652 10633 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_0.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_0.1557210500371863-231/raft_consensus-itest-cluster/ts-0/wal uuid: "cff628f906034bfbabca7dc6fad72eda" format_stamp: "Formatted at 2019-05-07 06:32:29 on dist-test-slave-dist-test-slave-rqp6" I0507 06:32:30.224061 10633 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_0.1557210500371863-231/raft_consensus-itest-cluster/ts-0/wal metadata directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_0.1557210500371863-231/raft_consensus-itest-cluster/ts-0/wal 1 data directories: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_0.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0507 06:32:30.305119 10633 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0507 06:32:30.318557 10633 ts_tablet_manager.cc:344] Loading tablet metadata (0/1 complete) I0507 06:32:30.324965 10633 ts_tablet_manager.cc:356] Loaded tablet metadata (1 total tablets, 1 live tablets) I0507 06:32:30.325057 10633 ts_tablet_manager.cc:362] Registering tablets (0/1 complete) I0507 06:32:30.328940 10633 ts_tablet_manager.cc:376] Registered 1 tablets I0507 06:32:30.329744 10657 tablet_bootstrap.cc:439] T 1cd3f33e70bc4d9897d06e1c7ae38792 P cff628f906034bfbabca7dc6fad72eda: Bootstrap starting. I0507 06:32:30.346921 10657 log.cc:525] T 1cd3f33e70bc4d9897d06e1c7ae38792 P cff628f906034bfbabca7dc6fad72eda: Log is configured to *not* fsync() on all Append() calls I0507 06:32:30.351774 10657 tablet_bootstrap.cc:439] T 1cd3f33e70bc4d9897d06e1c7ae38792 P cff628f906034bfbabca7dc6fad72eda: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I0507 06:32:30.352190 10657 tablet_bootstrap.cc:439] T 1cd3f33e70bc4d9897d06e1c7ae38792 P cff628f906034bfbabca7dc6fad72eda: Bootstrap complete. I0507 06:32:30.353322 10657 ts_tablet_manager.cc:1053] T 1cd3f33e70bc4d9897d06e1c7ae38792 P cff628f906034bfbabca7dc6fad72eda: Time spent bootstrapping tablet: real 0.024s user 0.013s sys 0.009s I0507 06:32:30.363128 10657 raft_consensus.cc:340] T 1cd3f33e70bc4d9897d06e1c7ae38792 P cff628f906034bfbabca7dc6fad72eda [term 1 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cff628f906034bfbabca7dc6fad72eda" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 45363 } } I0507 06:32:30.366540 10657 raft_consensus.cc:705] T 1cd3f33e70bc4d9897d06e1c7ae38792 P cff628f906034bfbabca7dc6fad72eda [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: cff628f906034bfbabca7dc6fad72eda, State: Initialized, Role: FOLLOWER I0507 06:32:30.367170 10657 consensus_queue.cc:229] T 1cd3f33e70bc4d9897d06e1c7ae38792 P cff628f906034bfbabca7dc6fad72eda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cff628f906034bfbabca7dc6fad72eda" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 45363 } } I0507 06:32:30.367393 10657 raft_consensus.cc:380] T 1cd3f33e70bc4d9897d06e1c7ae38792 P cff628f906034bfbabca7dc6fad72eda [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0507 06:32:30.367460 10657 raft_consensus.cc:472] T 1cd3f33e70bc4d9897d06e1c7ae38792 P cff628f906034bfbabca7dc6fad72eda [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0507 06:32:30.367576 10657 raft_consensus.cc:2886] T 1cd3f33e70bc4d9897d06e1c7ae38792 P cff628f906034bfbabca7dc6fad72eda [term 1 FOLLOWER]: Advancing to term 2 I0507 06:32:30.370568 10657 raft_consensus.cc:494] T 1cd3f33e70bc4d9897d06e1c7ae38792 P cff628f906034bfbabca7dc6fad72eda [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cff628f906034bfbabca7dc6fad72eda" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 45363 } } I0507 06:32:30.371022 10657 leader_election.cc:310] T 1cd3f33e70bc4d9897d06e1c7ae38792 P cff628f906034bfbabca7dc6fad72eda [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cff628f906034bfbabca7dc6fad72eda; no voters: I0507 06:32:30.381162 10657 leader_election.cc:296] T 1cd3f33e70bc4d9897d06e1c7ae38792 P cff628f906034bfbabca7dc6fad72eda [CANDIDATE]: Term 2 election: Requested vote from peers I0507 06:32:30.381340 10683 raft_consensus.cc:2640] T 1cd3f33e70bc4d9897d06e1c7ae38792 P cff628f906034bfbabca7dc6fad72eda [term 2 FOLLOWER]: Leader election won for term 2 I0507 06:32:30.382952 10657 ts_tablet_manager.cc:1081] T 1cd3f33e70bc4d9897d06e1c7ae38792 P cff628f906034bfbabca7dc6fad72eda: Time spent starting tablet: real 0.029s user 0.011s sys 0.011s I0507 06:32:30.384727 10683 raft_consensus.cc:667] T 1cd3f33e70bc4d9897d06e1c7ae38792 P cff628f906034bfbabca7dc6fad72eda [term 2 LEADER]: Becoming Leader. State: Replica: cff628f906034bfbabca7dc6fad72eda, State: Running, Role: LEADER I0507 06:32:30.385304 10683 consensus_queue.cc:206] T 1cd3f33e70bc4d9897d06e1c7ae38792 P cff628f906034bfbabca7dc6fad72eda [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cff628f906034bfbabca7dc6fad72eda" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 45363 } } I0507 06:32:30.440676 10633 rpc_server.cc:205] RPC server started. Bound to: 127.0.57.193:45363 I0507 06:32:30.441932 10633 webserver.cc:308] Webserver started at http://127.0.57.193:44209/ using document root and password file I0507 06:32:30.442631 10633 server_base.cc:608] Dumped server information to /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_0.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data/info.pb I0507 06:32:30.446880 231 external_mini_cluster.cc:995] Started /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver as pid 10633 I0507 06:32:30.456235 10768 heartbeater.cc:345] Connected to a master server at 127.0.57.254:39193 I0507 06:32:30.456463 10768 heartbeater.cc:426] Registering TS with master... I0507 06:32:30.457065 10768 heartbeater.cc:475] Master 127.0.57.254:39193 requested a full tablet report, sending... W0507 06:32:30.459426 10599 master_service.cc:205] Configuration error: replica replacement scheme (PREPARE_REPLACEMENT_BEFORE_EVICTION) of the tablet server cff628f906034bfbabca7dc6fad72eda at 127.0.57.193:40979 differs from the catalog manager's (EVICT_FIRST); they must be run with the same scheme (controlled by the --raft_prepare_replacement_before_eviction flag) W0507 06:32:30.460461 10768 heartbeater.cc:587] Failed to heartbeat to 127.0.57.254:39193 (0 consecutive failures): Configuration error: replica replacement scheme (PREPARE_REPLACEMENT_BEFORE_EVICTION) of the tablet server cff628f906034bfbabca7dc6fad72eda at 127.0.57.193:40979 differs from the catalog manager's (EVICT_FIRST); they must be run with the same scheme (controlled by the --raft_prepare_replacement_before_eviction flag) E0507 06:32:30.460577 10768 heartbeater.cc:622] master detected incompatibility: Configuration error: replica replacement scheme (PREPARE_REPLACEMENT_BEFORE_EVICTION) of the tablet server cff628f906034bfbabca7dc6fad72eda at 127.0.57.193:40979 differs from the catalog manager's (EVICT_FIRST); they must be run with the same scheme (controlled by the --raft_prepare_replacement_before_eviction flag) I0507 06:32:30.460651 10768 heartbeater.cc:426] Registering TS with master... I0507 06:32:30.460844 10768 heartbeater.cc:475] Master 127.0.57.254:39193 requested a full tablet report, sending... W0507 06:32:30.461670 10599 master_service.cc:205] Configuration error: replica replacement scheme (PREPARE_REPLACEMENT_BEFORE_EVICTION) of the tablet server cff628f906034bfbabca7dc6fad72eda at 127.0.57.193:40979 differs from the catalog manager's (EVICT_FIRST); they must be run with the same scheme (controlled by the --raft_prepare_replacement_before_eviction flag) I0507 06:32:30.462191 10768 heartbeater.cc:426] Registering TS with master... I0507 06:32:30.462370 10768 heartbeater.cc:475] Master 127.0.57.254:39193 requested a full tablet report, sending... W0507 06:32:30.463052 10599 master_service.cc:205] Configuration error: replica replacement scheme (PREPARE_REPLACEMENT_BEFORE_EVICTION) of the tablet server cff628f906034bfbabca7dc6fad72eda at 127.0.57.193:40979 differs from the catalog manager's (EVICT_FIRST); they must be run with the same scheme (controlled by the --raft_prepare_replacement_before_eviction flag) W0507 06:32:30.463477 10768 heartbeater.cc:380] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts. I0507 06:32:30.963723 10768 heartbeater.cc:426] Registering TS with master... I0507 06:32:30.964115 10768 heartbeater.cc:475] Master 127.0.57.254:39193 requested a full tablet report, sending... W0507 06:32:30.965554 10599 master_service.cc:205] Configuration error: replica replacement scheme (PREPARE_REPLACEMENT_BEFORE_EVICTION) of the tablet server cff628f906034bfbabca7dc6fad72eda at 127.0.57.193:40979 differs from the catalog manager's (EVICT_FIRST); they must be run with the same scheme (controlled by the --raft_prepare_replacement_before_eviction flag) I0507 06:32:31.466671 10768 heartbeater.cc:426] Registering TS with master... I0507 06:32:31.467070 10768 heartbeater.cc:475] Master 127.0.57.254:39193 requested a full tablet report, sending... W0507 06:32:31.468673 10599 master_service.cc:205] Configuration error: replica replacement scheme (PREPARE_REPLACEMENT_BEFORE_EVICTION) of the tablet server cff628f906034bfbabca7dc6fad72eda at 127.0.57.193:40979 differs from the catalog manager's (EVICT_FIRST); they must be run with the same scheme (controlled by the --raft_prepare_replacement_before_eviction flag) I0507 06:32:31.953809 231 external_mini_cluster.cc:1172] Attempting to check leaks for /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver pid 10633 I0507 06:32:31.961341 10765 generic_service.cc:171] Checking for leaks (request via RPC) I0507 06:32:32.011600 10768 heartbeater.cc:426] Registering TS with master... I0507 06:32:32.011993 10768 heartbeater.cc:475] Master 127.0.57.254:39193 requested a full tablet report, sending... I0507 06:32:32.013264 231 external_mini_cluster.cc:1131] Killing /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver with pid 10633 W0507 06:32:32.013593 10599 master_service.cc:205] Configuration error: replica replacement scheme (PREPARE_REPLACEMENT_BEFORE_EVICTION) of the tablet server cff628f906034bfbabca7dc6fad72eda at 127.0.57.193:40979 differs from the catalog manager's (EVICT_FIRST); they must be run with the same scheme (controlled by the --raft_prepare_replacement_before_eviction flag) W0507 06:32:32.026332 10580 connection.cc:647] server connection from 127.0.57.193:40979 recv error: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107) I0507 06:32:32.026883 231 external_mini_cluster.cc:933] Running /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_0.1557210500371863-231/raft_consensus-itest-cluster/ts-0/wal --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_0.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data --block_manager=log --rpc_bind_addresses=127.0.57.193:45363 --local_ip_for_outbound_sockets=127.0.57.193 --webserver_port=44209 --webserver_interface=127.0.57.193 --tserver_master_addrs=127.0.57.254:39193 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_0.1557210500371863-231/raft_consensus-itest-cluster/ts-0/logs --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_0.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --heartbeat_incompatible_replica_management_is_fatal=false --heartbeat_interval_ms=500 --raft_prepare_replacement_before_eviction=false --heartbeat_inject_required_feature_flag=999 with env {} WARNING: Logging before InitGoogleLogging() is written to STDERR W0507 06:32:32.200506 10776 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0507 06:32:32.200814 10776 flags.cc:404] Enabled unsafe flag: --heartbeat_inject_required_feature_flag=999 W0507 06:32:32.200856 10776 flags.cc:404] Enabled unsafe flag: --never_fsync=true W0507 06:32:32.202925 10776 flags.cc:404] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false W0507 06:32:32.202996 10776 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W0507 06:32:32.203106 10776 flags.cc:404] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.57.193 I0507 06:32:32.207417 10776 tablet_server_main.cc:78] Tablet server non-default flags: --raft_prepare_replacement_before_eviction=false --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_0.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_0.1557210500371863-231/raft_consensus-itest-cluster/ts-0/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.0.57.193:45363 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_0.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data/info.pb --webserver_interface=127.0.57.193 --webserver_port=44209 --heartbeat_incompatible_replica_management_is_fatal=false --heartbeat_inject_required_feature_flag=999 --heartbeat_interval_ms=500 --tserver_master_addrs=127.0.57.254:39193 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.0.57.193 --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_0.1557210500371863-231/raft_consensus-itest-cluster/ts-0/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.10.0-cdh6.x-SNAPSHOT revision 7d0ee26697926e87af9ef395008799577f6ab43f build type FASTDEBUG built by jenkins at 06 May 2019 22:44:21 PST on quasar-knmpfx-1.vpc.cloudera.com build id 322017 ASAN enabled I0507 06:32:32.215603 10776 system_ntp.cc:202] NTP initialized. Skew: 500ppm Current error: 150516us I0507 06:32:32.216313 10776 fs_manager.cc:263] Metadata directory not provided I0507 06:32:32.216390 10776 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0507 06:32:32.222410 10776 fs_manager.cc:399] Time spent opening directory manager: real 0.004s user 0.002s sys 0.002s I0507 06:32:32.223103 10776 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0507 06:32:32.223714 10776 file_cache.cc:466] Constructed file cache lbm with capacity 419430 I0507 06:32:32.226750 10776 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s I0507 06:32:32.226830 10776 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_0.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_0.1557210500371863-231/raft_consensus-itest-cluster/ts-0/wal uuid: "cff628f906034bfbabca7dc6fad72eda" format_stamp: "Formatted at 2019-05-07 06:32:29 on dist-test-slave-dist-test-slave-rqp6" I0507 06:32:32.227874 10776 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_0.1557210500371863-231/raft_consensus-itest-cluster/ts-0/wal metadata directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_0.1557210500371863-231/raft_consensus-itest-cluster/ts-0/wal 1 data directories: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_0.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0507 06:32:32.258255 10776 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0507 06:32:32.260212 10776 ts_tablet_manager.cc:344] Loading tablet metadata (0/1 complete) I0507 06:32:32.264345 10776 ts_tablet_manager.cc:356] Loaded tablet metadata (1 total tablets, 1 live tablets) I0507 06:32:32.264420 10776 ts_tablet_manager.cc:362] Registering tablets (0/1 complete) I0507 06:32:32.267161 10776 ts_tablet_manager.cc:376] Registered 1 tablets I0507 06:32:32.268064 10791 tablet_bootstrap.cc:439] T 1cd3f33e70bc4d9897d06e1c7ae38792 P cff628f906034bfbabca7dc6fad72eda: Bootstrap starting. I0507 06:32:32.288161 10791 log.cc:525] T 1cd3f33e70bc4d9897d06e1c7ae38792 P cff628f906034bfbabca7dc6fad72eda: Log is configured to *not* fsync() on all Append() calls I0507 06:32:32.292654 10791 tablet_bootstrap.cc:439] T 1cd3f33e70bc4d9897d06e1c7ae38792 P cff628f906034bfbabca7dc6fad72eda: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I0507 06:32:32.293540 10791 tablet_bootstrap.cc:439] T 1cd3f33e70bc4d9897d06e1c7ae38792 P cff628f906034bfbabca7dc6fad72eda: Bootstrap complete. I0507 06:32:32.294723 10791 ts_tablet_manager.cc:1053] T 1cd3f33e70bc4d9897d06e1c7ae38792 P cff628f906034bfbabca7dc6fad72eda: Time spent bootstrapping tablet: real 0.027s user 0.010s sys 0.015s I0507 06:32:32.305572 10776 rpc_server.cc:205] RPC server started. Bound to: 127.0.57.193:45363 I0507 06:32:32.307061 10776 webserver.cc:308] Webserver started at http://127.0.57.193:44209/ using document root and password file I0507 06:32:32.307870 10776 server_base.cc:608] Dumped server information to /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_0.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data/info.pb I0507 06:32:32.308931 231 external_mini_cluster.cc:995] Started /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver as pid 10776 I0507 06:32:32.311410 10791 raft_consensus.cc:340] T 1cd3f33e70bc4d9897d06e1c7ae38792 P cff628f906034bfbabca7dc6fad72eda [term 2 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cff628f906034bfbabca7dc6fad72eda" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 45363 } } I0507 06:32:32.314188 10791 raft_consensus.cc:705] T 1cd3f33e70bc4d9897d06e1c7ae38792 P cff628f906034bfbabca7dc6fad72eda [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: cff628f906034bfbabca7dc6fad72eda, State: Initialized, Role: FOLLOWER I0507 06:32:32.314684 10791 consensus_queue.cc:229] T 1cd3f33e70bc4d9897d06e1c7ae38792 P cff628f906034bfbabca7dc6fad72eda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cff628f906034bfbabca7dc6fad72eda" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 45363 } } I0507 06:32:32.314878 10791 raft_consensus.cc:380] T 1cd3f33e70bc4d9897d06e1c7ae38792 P cff628f906034bfbabca7dc6fad72eda [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0507 06:32:32.314949 10791 raft_consensus.cc:472] T 1cd3f33e70bc4d9897d06e1c7ae38792 P cff628f906034bfbabca7dc6fad72eda [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0507 06:32:32.315063 10791 raft_consensus.cc:2886] T 1cd3f33e70bc4d9897d06e1c7ae38792 P cff628f906034bfbabca7dc6fad72eda [term 2 FOLLOWER]: Advancing to term 3 I0507 06:32:32.318140 10791 raft_consensus.cc:494] T 1cd3f33e70bc4d9897d06e1c7ae38792 P cff628f906034bfbabca7dc6fad72eda [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cff628f906034bfbabca7dc6fad72eda" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 45363 } } I0507 06:32:32.318593 10791 leader_election.cc:310] T 1cd3f33e70bc4d9897d06e1c7ae38792 P cff628f906034bfbabca7dc6fad72eda [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cff628f906034bfbabca7dc6fad72eda; no voters: I0507 06:32:32.319447 10791 leader_election.cc:296] T 1cd3f33e70bc4d9897d06e1c7ae38792 P cff628f906034bfbabca7dc6fad72eda [CANDIDATE]: Term 3 election: Requested vote from peers I0507 06:32:32.319571 10900 raft_consensus.cc:2640] T 1cd3f33e70bc4d9897d06e1c7ae38792 P cff628f906034bfbabca7dc6fad72eda [term 3 FOLLOWER]: Leader election won for term 3 I0507 06:32:32.321029 10900 raft_consensus.cc:667] T 1cd3f33e70bc4d9897d06e1c7ae38792 P cff628f906034bfbabca7dc6fad72eda [term 3 LEADER]: Becoming Leader. State: Replica: cff628f906034bfbabca7dc6fad72eda, State: Running, Role: LEADER I0507 06:32:32.321259 10791 ts_tablet_manager.cc:1081] T 1cd3f33e70bc4d9897d06e1c7ae38792 P cff628f906034bfbabca7dc6fad72eda: Time spent starting tablet: real 0.026s user 0.016s sys 0.007s I0507 06:32:32.321568 10900 consensus_queue.cc:206] T 1cd3f33e70bc4d9897d06e1c7ae38792 P cff628f906034bfbabca7dc6fad72eda [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cff628f906034bfbabca7dc6fad72eda" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 45363 } } I0507 06:32:32.325294 10897 heartbeater.cc:345] Connected to a master server at 127.0.57.254:39193 I0507 06:32:32.325520 10897 heartbeater.cc:426] Registering TS with master... I0507 06:32:32.326063 10897 heartbeater.cc:475] Master 127.0.57.254:39193 requested a full tablet report, sending... W0507 06:32:32.327316 10581 messenger.cc:389] Unable to handle RPC call: Not implemented: call requires unsupported application feature flags: 999 W0507 06:32:32.327894 10897 heartbeater.cc:587] Failed to heartbeat to 127.0.57.254:39193 (0 consecutive failures): Remote error: Failed to send heartbeat to master: unsupported feature flags E0507 06:32:32.328016 10897 heartbeater.cc:622] master does not support required feature flags: Remote error: Failed to send heartbeat to master: unsupported feature flags I0507 06:32:32.328081 10897 heartbeater.cc:426] Registering TS with master... I0507 06:32:32.328235 10897 heartbeater.cc:475] Master 127.0.57.254:39193 requested a full tablet report, sending... W0507 06:32:32.328840 10581 messenger.cc:389] Unable to handle RPC call: Not implemented: call requires unsupported application feature flags: 999 I0507 06:32:32.329207 10897 heartbeater.cc:426] Registering TS with master... I0507 06:32:32.329388 10897 heartbeater.cc:475] Master 127.0.57.254:39193 requested a full tablet report, sending... W0507 06:32:32.329885 10581 messenger.cc:389] Unable to handle RPC call: Not implemented: call requires unsupported application feature flags: 999 W0507 06:32:32.330229 10897 heartbeater.cc:380] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts. I0507 06:32:32.830446 10897 heartbeater.cc:426] Registering TS with master... I0507 06:32:32.830796 10897 heartbeater.cc:475] Master 127.0.57.254:39193 requested a full tablet report, sending... W0507 06:32:32.832043 10581 messenger.cc:389] Unable to handle RPC call: Not implemented: call requires unsupported application feature flags: 999 I0507 06:32:33.332799 10897 heartbeater.cc:426] Registering TS with master... I0507 06:32:33.333171 10897 heartbeater.cc:475] Master 127.0.57.254:39193 requested a full tablet report, sending... W0507 06:32:33.334313 10581 messenger.cc:389] Unable to handle RPC call: Not implemented: call requires unsupported application feature flags: 999 I0507 06:32:33.812211 231 external_mini_cluster.cc:1172] Attempting to check leaks for /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver pid 10776 I0507 06:32:33.820485 10894 generic_service.cc:171] Checking for leaks (request via RPC) I0507 06:32:33.869587 10897 heartbeater.cc:426] Registering TS with master... I0507 06:32:33.870016 10897 heartbeater.cc:475] Master 127.0.57.254:39193 requested a full tablet report, sending... I0507 06:32:33.870932 231 external_mini_cluster.cc:1131] Killing /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver with pid 10776 W0507 06:32:33.871104 10581 messenger.cc:389] Unable to handle RPC call: Not implemented: call requires unsupported application feature flags: 999 W0507 06:32:33.883177 10581 connection.cc:647] server connection from 127.0.57.193:33961 recv error: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107) I0507 06:32:33.883507 231 external_mini_cluster.cc:1172] Attempting to check leaks for /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-master pid 10573 I0507 06:32:33.884738 10629 generic_service.cc:171] Checking for leaks (request via RPC) I0507 06:32:33.955057 231 external_mini_cluster.cc:1131] Killing /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-master with pid 10573 [ OK ] IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig/0 (5268 ms) [ RUN ] IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig/1 I0507 06:32:33.976763 231 test_util.cc:212] Using random seed: 1641335195 I0507 06:32:33.978236 231 ts_itest-base.cc:112] Starting cluster with: I0507 06:32:33.978286 231 ts_itest-base.cc:113] -------------- I0507 06:32:33.978302 231 ts_itest-base.cc:114] 1 tablet servers I0507 06:32:33.978338 231 ts_itest-base.cc:115] 1 replicas per TS I0507 06:32:33.978353 231 ts_itest-base.cc:116] -------------- I0507 06:32:33.979413 231 external_mini_cluster.cc:933] Running /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-master /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-master --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_1.1557210500371863-231/raft_consensus-itest-cluster/master-0/wal --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_1.1557210500371863-231/raft_consensus-itest-cluster/master-0/data --block_manager=log --webserver_interface=localhost --ipki_ca_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.0.57.254:39017 --webserver_interface=127.0.57.254 --webserver_port=0 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_1.1557210500371863-231/raft_consensus-itest-cluster/master-0/logs --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_1.1557210500371863-231/raft_consensus-itest-cluster/master-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=true --rpc_reuseport=true with env {} WARNING: Logging before InitGoogleLogging() is written to STDERR W0507 06:32:34.182739 10914 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0507 06:32:34.182971 10914 flags.cc:404] Enabled unsafe flag: --never_fsync=true W0507 06:32:34.185344 10914 flags.cc:404] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true W0507 06:32:34.185407 10914 flags.cc:404] Enabled experimental flag: --ipki_ca_key_size=1024 W0507 06:32:34.185436 10914 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W0507 06:32:34.185472 10914 flags.cc:404] Enabled experimental flag: --tsk_num_rsa_bits=512 W0507 06:32:34.185505 10914 flags.cc:404] Enabled experimental flag: --rpc_reuseport=true I0507 06:32:34.190083 10914 master_main.cc:73] Master server non-default flags: --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_1.1557210500371863-231/raft_consensus-itest-cluster/master-0/data --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_1.1557210500371863-231/raft_consensus-itest-cluster/master-0/wal --ipki_ca_key_size=1024 --ipki_server_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.0.57.254:39017 --rpc_reuseport=true --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_1.1557210500371863-231/raft_consensus-itest-cluster/master-0/data/info.pb --webserver_interface=127.0.57.254 --webserver_port=0 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_1.1557210500371863-231/raft_consensus-itest-cluster/master-0/logs --logbuflevel=-1 --logtostderr=true Master server version: kudu 1.10.0-cdh6.x-SNAPSHOT revision 7d0ee26697926e87af9ef395008799577f6ab43f build type FASTDEBUG built by jenkins at 06 May 2019 22:44:21 PST on quasar-knmpfx-1.vpc.cloudera.com build id 322017 ASAN enabled I0507 06:32:34.196363 10914 system_ntp.cc:202] NTP initialized. Skew: 500ppm Current error: 151516us I0507 06:32:34.197657 10914 fs_manager.cc:263] Metadata directory not provided I0507 06:32:34.197748 10914 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0507 06:32:34.198055 10914 server_base.cc:433] Could not load existing FS layout: Not found: could not find a healthy instance file I0507 06:32:34.198150 10914 server_base.cc:434] Attempting to create new FS layout instead I0507 06:32:34.200745 10914 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_1.1557210500371863-231/raft_consensus-itest-cluster/master-0/data/instance: uuid: "9a3648e904f94061bc160bfd15e91a51" format_stamp: "Formatted at 2019-05-07 06:32:34 on dist-test-slave-dist-test-slave-rqp6" I0507 06:32:34.201294 10914 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_1.1557210500371863-231/raft_consensus-itest-cluster/master-0/wal/instance: uuid: "9a3648e904f94061bc160bfd15e91a51" format_stamp: "Formatted at 2019-05-07 06:32:34 on dist-test-slave-dist-test-slave-rqp6" I0507 06:32:34.208361 10914 fs_manager.cc:503] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s I0507 06:32:34.209475 10914 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0507 06:32:34.210049 10914 file_cache.cc:466] Constructed file cache lbm with capacity 419430 I0507 06:32:34.213538 10914 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s I0507 06:32:34.213629 10914 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_1.1557210500371863-231/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_1.1557210500371863-231/raft_consensus-itest-cluster/master-0/wal uuid: "9a3648e904f94061bc160bfd15e91a51" format_stamp: "Formatted at 2019-05-07 06:32:34 on dist-test-slave-dist-test-slave-rqp6" I0507 06:32:34.213766 10914 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_1.1557210500371863-231/raft_consensus-itest-cluster/master-0/wal metadata directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_1.1557210500371863-231/raft_consensus-itest-cluster/master-0/wal 1 data directories: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_1.1557210500371863-231/raft_consensus-itest-cluster/master-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0507 06:32:34.265547 10914 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0507 06:32:34.281687 10914 rpc_server.cc:205] RPC server started. Bound to: 127.0.57.254:39017 I0507 06:32:34.282995 10914 webserver.cc:308] Webserver started at http://127.0.57.254:38967/ using document root and password file I0507 06:32:34.283836 10914 server_base.cc:608] Dumped server information to /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_1.1557210500371863-231/raft_consensus-itest-cluster/master-0/data/info.pb I0507 06:32:34.287776 10973 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0507 06:32:34.292227 231 external_mini_cluster.cc:995] Started /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-master as pid 10914 I0507 06:32:34.292954 231 external_mini_cluster.cc:933] Running /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_1.1557210500371863-231/raft_consensus-itest-cluster/ts-0/wal --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_1.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data --block_manager=log --rpc_bind_addresses=127.0.57.193:0 --local_ip_for_outbound_sockets=127.0.57.193 --webserver_interface=127.0.57.193 --webserver_port=0 --tserver_master_addrs=127.0.57.254:39017 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_1.1557210500371863-231/raft_consensus-itest-cluster/ts-0/logs --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_1.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --heartbeat_incompatible_replica_management_is_fatal=false --heartbeat_interval_ms=500 --raft_prepare_replacement_before_eviction=true with env {} I0507 06:32:34.300163 10973 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 9a3648e904f94061bc160bfd15e91a51: Bootstrap starting. I0507 06:32:34.303519 10973 tablet_bootstrap.cc:590] T 00000000000000000000000000000000 P 9a3648e904f94061bc160bfd15e91a51: No blocks or log segments found. Creating new log. I0507 06:32:34.304677 10973 log.cc:525] T 00000000000000000000000000000000 P 9a3648e904f94061bc160bfd15e91a51: Log is configured to *not* fsync() on all Append() calls I0507 06:32:34.306838 10973 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 9a3648e904f94061bc160bfd15e91a51: No bootstrap required, opened a new log I0507 06:32:34.314304 10973 raft_consensus.cc:340] T 00000000000000000000000000000000 P 9a3648e904f94061bc160bfd15e91a51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a3648e904f94061bc160bfd15e91a51" member_type: VOTER } I0507 06:32:34.314594 10973 raft_consensus.cc:366] T 00000000000000000000000000000000 P 9a3648e904f94061bc160bfd15e91a51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0507 06:32:34.314662 10973 raft_consensus.cc:705] T 00000000000000000000000000000000 P 9a3648e904f94061bc160bfd15e91a51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a3648e904f94061bc160bfd15e91a51, State: Initialized, Role: FOLLOWER I0507 06:32:34.315299 10973 consensus_queue.cc:229] T 00000000000000000000000000000000 P 9a3648e904f94061bc160bfd15e91a51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a3648e904f94061bc160bfd15e91a51" member_type: VOTER } I0507 06:32:34.315502 10973 raft_consensus.cc:380] T 00000000000000000000000000000000 P 9a3648e904f94061bc160bfd15e91a51 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0507 06:32:34.315589 10973 raft_consensus.cc:472] T 00000000000000000000000000000000 P 9a3648e904f94061bc160bfd15e91a51 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0507 06:32:34.315732 10973 raft_consensus.cc:2886] T 00000000000000000000000000000000 P 9a3648e904f94061bc160bfd15e91a51 [term 0 FOLLOWER]: Advancing to term 1 I0507 06:32:34.317631 10973 raft_consensus.cc:494] T 00000000000000000000000000000000 P 9a3648e904f94061bc160bfd15e91a51 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a3648e904f94061bc160bfd15e91a51" member_type: VOTER } I0507 06:32:34.318145 10973 leader_election.cc:310] T 00000000000000000000000000000000 P 9a3648e904f94061bc160bfd15e91a51 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9a3648e904f94061bc160bfd15e91a51; no voters: I0507 06:32:34.318934 10973 leader_election.cc:296] T 00000000000000000000000000000000 P 9a3648e904f94061bc160bfd15e91a51 [CANDIDATE]: Term 1 election: Requested vote from peers I0507 06:32:34.319352 10976 raft_consensus.cc:2640] T 00000000000000000000000000000000 P 9a3648e904f94061bc160bfd15e91a51 [term 1 FOLLOWER]: Leader election won for term 1 I0507 06:32:34.320168 10976 raft_consensus.cc:667] T 00000000000000000000000000000000 P 9a3648e904f94061bc160bfd15e91a51 [term 1 LEADER]: Becoming Leader. State: Replica: 9a3648e904f94061bc160bfd15e91a51, State: Running, Role: LEADER I0507 06:32:34.320699 10976 consensus_queue.cc:206] T 00000000000000000000000000000000 P 9a3648e904f94061bc160bfd15e91a51 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a3648e904f94061bc160bfd15e91a51" member_type: VOTER } I0507 06:32:34.323680 10973 sys_catalog.cc:337] T 00000000000000000000000000000000 P 9a3648e904f94061bc160bfd15e91a51 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 1 leader_uuid: "9a3648e904f94061bc160bfd15e91a51" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a3648e904f94061bc160bfd15e91a51" member_type: VOTER } } I0507 06:32:34.323948 10973 sys_catalog.cc:340] T 00000000000000000000000000000000 P 9a3648e904f94061bc160bfd15e91a51 [sys.catalog]: This master's current role is: LEADER I0507 06:32:34.323926 10977 sys_catalog.cc:337] T 00000000000000000000000000000000 P 9a3648e904f94061bc160bfd15e91a51 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9a3648e904f94061bc160bfd15e91a51" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a3648e904f94061bc160bfd15e91a51" member_type: VOTER } } I0507 06:32:34.323673 10978 sys_catalog.cc:337] T 00000000000000000000000000000000 P 9a3648e904f94061bc160bfd15e91a51 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9a3648e904f94061bc160bfd15e91a51. Latest consensus state: current_term: 1 leader_uuid: "9a3648e904f94061bc160bfd15e91a51" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a3648e904f94061bc160bfd15e91a51" member_type: VOTER } } I0507 06:32:34.324240 10977 sys_catalog.cc:340] T 00000000000000000000000000000000 P 9a3648e904f94061bc160bfd15e91a51 [sys.catalog]: This master's current role is: LEADER I0507 06:32:34.324280 10978 sys_catalog.cc:340] T 00000000000000000000000000000000 P 9a3648e904f94061bc160bfd15e91a51 [sys.catalog]: This master's current role is: LEADER I0507 06:32:34.326426 10973 sys_catalog.cc:424] T 00000000000000000000000000000000 P 9a3648e904f94061bc160bfd15e91a51 [sys.catalog]: configured and running, proceeding with master startup. I0507 06:32:34.326545 10980 catalog_manager.cc:1055] Loading table and tablet metadata into memory... I0507 06:32:34.330793 10980 catalog_manager.cc:1066] Initializing Kudu internal certificate authority... I0507 06:32:34.374438 10980 catalog_manager.cc:950] Generated new certificate authority record I0507 06:32:34.375849 10980 catalog_manager.cc:1075] Loading token signing keys... I0507 06:32:34.387892 10980 catalog_manager.cc:4292] T 00000000000000000000000000000000 P 9a3648e904f94061bc160bfd15e91a51: Generated new TSK 0 WARNING: Logging before InitGoogleLogging() is written to STDERR W0507 06:32:34.460580 10974 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0507 06:32:34.460886 10974 flags.cc:404] Enabled unsafe flag: --never_fsync=true W0507 06:32:34.463615 10974 flags.cc:404] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true W0507 06:32:34.463686 10974 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W0507 06:32:34.463825 10974 flags.cc:404] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.57.193 I0507 06:32:34.480991 10974 tablet_server_main.cc:78] Tablet server non-default flags: --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_1.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_1.1557210500371863-231/raft_consensus-itest-cluster/ts-0/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.0.57.193:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_1.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data/info.pb --webserver_interface=127.0.57.193 --webserver_port=0 --heartbeat_incompatible_replica_management_is_fatal=false --heartbeat_interval_ms=500 --tserver_master_addrs=127.0.57.254:39017 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.0.57.193 --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_1.1557210500371863-231/raft_consensus-itest-cluster/ts-0/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.10.0-cdh6.x-SNAPSHOT revision 7d0ee26697926e87af9ef395008799577f6ab43f build type FASTDEBUG built by jenkins at 06 May 2019 22:44:21 PST on quasar-knmpfx-1.vpc.cloudera.com build id 322017 ASAN enabled I0507 06:32:34.489586 10974 system_ntp.cc:202] NTP initialized. Skew: 500ppm Current error: 151516us I0507 06:32:34.490411 10974 fs_manager.cc:263] Metadata directory not provided I0507 06:32:34.490476 10974 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0507 06:32:34.490778 10974 server_base.cc:433] Could not load existing FS layout: Not found: could not find a healthy instance file I0507 06:32:34.490855 10974 server_base.cc:434] Attempting to create new FS layout instead I0507 06:32:34.493515 10974 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_1.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data/instance: uuid: "b107758704444e01b3369a5f13ccff50" format_stamp: "Formatted at 2019-05-07 06:32:34 on dist-test-slave-dist-test-slave-rqp6" I0507 06:32:34.494124 10974 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_1.1557210500371863-231/raft_consensus-itest-cluster/ts-0/wal/instance: uuid: "b107758704444e01b3369a5f13ccff50" format_stamp: "Formatted at 2019-05-07 06:32:34 on dist-test-slave-dist-test-slave-rqp6" I0507 06:32:34.511540 10974 fs_manager.cc:503] Time spent creating directory manager: real 0.017s user 0.008s sys 0.008s I0507 06:32:34.512923 10974 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0507 06:32:34.513571 10974 file_cache.cc:466] Constructed file cache lbm with capacity 419430 I0507 06:32:34.517055 10974 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.000s sys 0.000s I0507 06:32:34.517150 10974 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_1.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_1.1557210500371863-231/raft_consensus-itest-cluster/ts-0/wal uuid: "b107758704444e01b3369a5f13ccff50" format_stamp: "Formatted at 2019-05-07 06:32:34 on dist-test-slave-dist-test-slave-rqp6" I0507 06:32:34.517349 10974 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_1.1557210500371863-231/raft_consensus-itest-cluster/ts-0/wal metadata directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_1.1557210500371863-231/raft_consensus-itest-cluster/ts-0/wal 1 data directories: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_1.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0507 06:32:34.565788 10974 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0507 06:32:34.567620 10974 ts_tablet_manager.cc:356] Loaded tablet metadata (0 total tablets, 0 live tablets) I0507 06:32:34.567701 10974 ts_tablet_manager.cc:376] Registered 0 tablets I0507 06:32:34.629307 10974 rpc_server.cc:205] RPC server started. Bound to: 127.0.57.193:42391 I0507 06:32:34.630478 10974 webserver.cc:308] Webserver started at http://127.0.57.193:41695/ using document root and password file I0507 06:32:34.631212 10974 server_base.cc:608] Dumped server information to /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_1.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data/info.pb I0507 06:32:34.635190 231 external_mini_cluster.cc:995] Started /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver as pid 10974 I0507 06:32:34.647368 11103 heartbeater.cc:345] Connected to a master server at 127.0.57.254:39017 I0507 06:32:34.647560 11103 heartbeater.cc:426] Registering TS with master... I0507 06:32:34.648092 11103 heartbeater.cc:475] Master 127.0.57.254:39017 requested a full tablet report, sending... I0507 06:32:34.649579 10940 ts_manager.cc:159] Registered new tserver with Master: b107758704444e01b3369a5f13ccff50 (127.0.57.193:42391) I0507 06:32:34.650224 231 external_mini_cluster.cc:623] 1 TS(s) registered with all masters I0507 06:32:34.651305 10940 master_service.cc:273] Signed X509 certificate for tserver {username='slave'} at 127.0.57.193:49095 I0507 06:32:34.664662 10940 catalog_manager.cc:1415] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54818: name: "TestTable" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 1 split_rows_range_bounds { } partition_schema { range_schema { columns { name: "key" } } } I0507 06:32:34.685415 11040 tablet_service.cc:1016] Processing CreateTablet for tablet 48ead67853264e088ba5bb88dcf815b0 (table=TestTable [id=45840a89cf7145a0bc237c16c9cf54ee]), partition=RANGE (key) PARTITION UNBOUNDED I0507 06:32:34.686280 11040 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet 48ead67853264e088ba5bb88dcf815b0. 1 dirs total, 0 dirs full, 0 dirs failed I0507 06:32:34.695976 11116 tablet_bootstrap.cc:439] T 48ead67853264e088ba5bb88dcf815b0 P b107758704444e01b3369a5f13ccff50: Bootstrap starting. I0507 06:32:34.698303 11116 tablet_bootstrap.cc:590] T 48ead67853264e088ba5bb88dcf815b0 P b107758704444e01b3369a5f13ccff50: No blocks or log segments found. Creating new log. I0507 06:32:34.699218 11116 log.cc:525] T 48ead67853264e088ba5bb88dcf815b0 P b107758704444e01b3369a5f13ccff50: Log is configured to *not* fsync() on all Append() calls I0507 06:32:34.701405 11116 tablet_bootstrap.cc:439] T 48ead67853264e088ba5bb88dcf815b0 P b107758704444e01b3369a5f13ccff50: No bootstrap required, opened a new log I0507 06:32:34.701650 11116 ts_tablet_manager.cc:1053] T 48ead67853264e088ba5bb88dcf815b0 P b107758704444e01b3369a5f13ccff50: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s I0507 06:32:34.707208 11116 raft_consensus.cc:340] T 48ead67853264e088ba5bb88dcf815b0 P b107758704444e01b3369a5f13ccff50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b107758704444e01b3369a5f13ccff50" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 42391 } } I0507 06:32:34.707543 11116 raft_consensus.cc:366] T 48ead67853264e088ba5bb88dcf815b0 P b107758704444e01b3369a5f13ccff50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0507 06:32:34.707604 11116 raft_consensus.cc:705] T 48ead67853264e088ba5bb88dcf815b0 P b107758704444e01b3369a5f13ccff50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b107758704444e01b3369a5f13ccff50, State: Initialized, Role: FOLLOWER I0507 06:32:34.708073 11116 consensus_queue.cc:229] T 48ead67853264e088ba5bb88dcf815b0 P b107758704444e01b3369a5f13ccff50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b107758704444e01b3369a5f13ccff50" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 42391 } } I0507 06:32:34.708310 11116 raft_consensus.cc:380] T 48ead67853264e088ba5bb88dcf815b0 P b107758704444e01b3369a5f13ccff50 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0507 06:32:34.708398 11116 raft_consensus.cc:472] T 48ead67853264e088ba5bb88dcf815b0 P b107758704444e01b3369a5f13ccff50 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0507 06:32:34.708498 11116 raft_consensus.cc:2886] T 48ead67853264e088ba5bb88dcf815b0 P b107758704444e01b3369a5f13ccff50 [term 0 FOLLOWER]: Advancing to term 1 I0507 06:32:34.710184 11116 raft_consensus.cc:494] T 48ead67853264e088ba5bb88dcf815b0 P b107758704444e01b3369a5f13ccff50 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b107758704444e01b3369a5f13ccff50" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 42391 } } I0507 06:32:34.710575 11116 leader_election.cc:310] T 48ead67853264e088ba5bb88dcf815b0 P b107758704444e01b3369a5f13ccff50 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b107758704444e01b3369a5f13ccff50; no voters: I0507 06:32:34.711151 11116 leader_election.cc:296] T 48ead67853264e088ba5bb88dcf815b0 P b107758704444e01b3369a5f13ccff50 [CANDIDATE]: Term 1 election: Requested vote from peers I0507 06:32:34.711304 11118 raft_consensus.cc:2640] T 48ead67853264e088ba5bb88dcf815b0 P b107758704444e01b3369a5f13ccff50 [term 1 FOLLOWER]: Leader election won for term 1 I0507 06:32:34.711983 11103 heartbeater.cc:467] Master 127.0.57.254:39017 was elected leader, sending a full tablet report... I0507 06:32:34.712081 11118 raft_consensus.cc:667] T 48ead67853264e088ba5bb88dcf815b0 P b107758704444e01b3369a5f13ccff50 [term 1 LEADER]: Becoming Leader. State: Replica: b107758704444e01b3369a5f13ccff50, State: Running, Role: LEADER I0507 06:32:34.712386 11116 ts_tablet_manager.cc:1081] T 48ead67853264e088ba5bb88dcf815b0 P b107758704444e01b3369a5f13ccff50: Time spent starting tablet: real 0.011s user 0.001s sys 0.010s I0507 06:32:34.712442 11118 consensus_queue.cc:206] T 48ead67853264e088ba5bb88dcf815b0 P b107758704444e01b3369a5f13ccff50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b107758704444e01b3369a5f13ccff50" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 42391 } } I0507 06:32:34.717357 10940 catalog_manager.cc:3988] T 48ead67853264e088ba5bb88dcf815b0 P b107758704444e01b3369a5f13ccff50 reported cstate change: term changed from 0 to 1, leader changed from to b107758704444e01b3369a5f13ccff50 (127.0.57.193). New cstate: current_term: 1 leader_uuid: "b107758704444e01b3369a5f13ccff50" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b107758704444e01b3369a5f13ccff50" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 42391 } health_report { overall_health: HEALTHY } } } I0507 06:32:34.730938 231 external_mini_cluster.cc:623] 1 TS(s) registered with all masters I0507 06:32:34.732340 231 ts_itest-base.cc:232] Waiting for 1 tablets on tserver b107758704444e01b3369a5f13ccff50 to finish bootstrapping I0507 06:32:34.739910 231 external_mini_cluster.cc:1172] Attempting to check leaks for /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver pid 10974 I0507 06:32:34.740553 11100 generic_service.cc:171] Checking for leaks (request via RPC) I0507 06:32:34.795248 231 external_mini_cluster.cc:1131] Killing /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver with pid 10974 I0507 06:32:34.806805 231 external_mini_cluster.cc:1172] Attempting to check leaks for /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-master pid 10914 I0507 06:32:34.807747 10970 generic_service.cc:171] Checking for leaks (request via RPC) I0507 06:32:34.881551 231 external_mini_cluster.cc:1131] Killing /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-master with pid 10914 I0507 06:32:34.897019 231 external_mini_cluster.cc:933] Running /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-master /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-master --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_1.1557210500371863-231/raft_consensus-itest-cluster/master-0/wal --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_1.1557210500371863-231/raft_consensus-itest-cluster/master-0/data --block_manager=log --webserver_interface=localhost --ipki_ca_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.0.57.254:39017 --webserver_interface=127.0.57.254 --webserver_port=38967 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_1.1557210500371863-231/raft_consensus-itest-cluster/master-0/logs --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_1.1557210500371863-231/raft_consensus-itest-cluster/master-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=true --rpc_reuseport=true with env {} WARNING: Logging before InitGoogleLogging() is written to STDERR W0507 06:32:35.103371 11125 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0507 06:32:35.103605 11125 flags.cc:404] Enabled unsafe flag: --never_fsync=true W0507 06:32:35.106279 11125 flags.cc:404] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true W0507 06:32:35.106348 11125 flags.cc:404] Enabled experimental flag: --ipki_ca_key_size=1024 W0507 06:32:35.106389 11125 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W0507 06:32:35.106425 11125 flags.cc:404] Enabled experimental flag: --tsk_num_rsa_bits=512 W0507 06:32:35.106467 11125 flags.cc:404] Enabled experimental flag: --rpc_reuseport=true I0507 06:32:35.111618 11125 master_main.cc:73] Master server non-default flags: --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_1.1557210500371863-231/raft_consensus-itest-cluster/master-0/data --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_1.1557210500371863-231/raft_consensus-itest-cluster/master-0/wal --ipki_ca_key_size=1024 --ipki_server_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.0.57.254:39017 --rpc_reuseport=true --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_1.1557210500371863-231/raft_consensus-itest-cluster/master-0/data/info.pb --webserver_interface=127.0.57.254 --webserver_port=38967 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_1.1557210500371863-231/raft_consensus-itest-cluster/master-0/logs --logbuflevel=-1 --logtostderr=true Master server version: kudu 1.10.0-cdh6.x-SNAPSHOT revision 7d0ee26697926e87af9ef395008799577f6ab43f build type FASTDEBUG built by jenkins at 06 May 2019 22:44:21 PST on quasar-knmpfx-1.vpc.cloudera.com build id 322017 ASAN enabled I0507 06:32:35.119154 11125 system_ntp.cc:202] NTP initialized. Skew: 500ppm Current error: 152016us I0507 06:32:35.120703 11125 fs_manager.cc:263] Metadata directory not provided I0507 06:32:35.120815 11125 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0507 06:32:35.127792 11125 fs_manager.cc:399] Time spent opening directory manager: real 0.005s user 0.005s sys 0.000s I0507 06:32:35.128587 11125 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0507 06:32:35.129252 11125 file_cache.cc:466] Constructed file cache lbm with capacity 419430 I0507 06:32:35.132901 11125 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s I0507 06:32:35.133008 11125 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_1.1557210500371863-231/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_1.1557210500371863-231/raft_consensus-itest-cluster/master-0/wal uuid: "9a3648e904f94061bc160bfd15e91a51" format_stamp: "Formatted at 2019-05-07 06:32:34 on dist-test-slave-dist-test-slave-rqp6" I0507 06:32:35.134143 11125 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_1.1557210500371863-231/raft_consensus-itest-cluster/master-0/wal metadata directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_1.1557210500371863-231/raft_consensus-itest-cluster/master-0/wal 1 data directories: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_1.1557210500371863-231/raft_consensus-itest-cluster/master-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0507 06:32:35.165151 11125 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0507 06:32:35.180263 11125 rpc_server.cc:205] RPC server started. Bound to: 127.0.57.254:39017 I0507 06:32:35.181608 11125 webserver.cc:308] Webserver started at http://127.0.57.254:38967/ using document root and password file I0507 06:32:35.182546 11125 server_base.cc:608] Dumped server information to /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_1.1557210500371863-231/raft_consensus-itest-cluster/master-0/data/info.pb I0507 06:32:35.190186 231 external_mini_cluster.cc:995] Started /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-master as pid 11125 I0507 06:32:35.190732 231 external_mini_cluster.cc:933] Running /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_1.1557210500371863-231/raft_consensus-itest-cluster/ts-0/wal --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_1.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data --block_manager=log --rpc_bind_addresses=127.0.57.193:42391 --local_ip_for_outbound_sockets=127.0.57.193 --webserver_port=41695 --webserver_interface=127.0.57.193 --tserver_master_addrs=127.0.57.254:39017 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_1.1557210500371863-231/raft_consensus-itest-cluster/ts-0/logs --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_1.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --heartbeat_incompatible_replica_management_is_fatal=false --heartbeat_interval_ms=500 --raft_prepare_replacement_before_eviction=true --raft_prepare_replacement_before_eviction=false with env {} I0507 06:32:35.191215 11184 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 9a3648e904f94061bc160bfd15e91a51: Bootstrap starting. I0507 06:32:35.211983 11184 log.cc:525] T 00000000000000000000000000000000 P 9a3648e904f94061bc160bfd15e91a51: Log is configured to *not* fsync() on all Append() calls I0507 06:32:35.226770 11184 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 9a3648e904f94061bc160bfd15e91a51: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=4 ignored=0} mutations{seen=2 ignored=0} orphaned_commits=0. Pending: 0 replicates I0507 06:32:35.227285 11184 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 9a3648e904f94061bc160bfd15e91a51: Bootstrap complete. I0507 06:32:35.237748 11184 raft_consensus.cc:340] T 00000000000000000000000000000000 P 9a3648e904f94061bc160bfd15e91a51 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a3648e904f94061bc160bfd15e91a51" member_type: VOTER } I0507 06:32:35.238543 11184 raft_consensus.cc:705] T 00000000000000000000000000000000 P 9a3648e904f94061bc160bfd15e91a51 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a3648e904f94061bc160bfd15e91a51, State: Initialized, Role: FOLLOWER I0507 06:32:35.239214 11184 consensus_queue.cc:229] T 00000000000000000000000000000000 P 9a3648e904f94061bc160bfd15e91a51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a3648e904f94061bc160bfd15e91a51" member_type: VOTER } I0507 06:32:35.239393 11184 raft_consensus.cc:380] T 00000000000000000000000000000000 P 9a3648e904f94061bc160bfd15e91a51 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0507 06:32:35.239481 11184 raft_consensus.cc:472] T 00000000000000000000000000000000 P 9a3648e904f94061bc160bfd15e91a51 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0507 06:32:35.239588 11184 raft_consensus.cc:2886] T 00000000000000000000000000000000 P 9a3648e904f94061bc160bfd15e91a51 [term 1 FOLLOWER]: Advancing to term 2 I0507 06:32:35.244122 11184 raft_consensus.cc:494] T 00000000000000000000000000000000 P 9a3648e904f94061bc160bfd15e91a51 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a3648e904f94061bc160bfd15e91a51" member_type: VOTER } I0507 06:32:35.244690 11184 leader_election.cc:310] T 00000000000000000000000000000000 P 9a3648e904f94061bc160bfd15e91a51 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9a3648e904f94061bc160bfd15e91a51; no voters: I0507 06:32:35.245474 11184 leader_election.cc:296] T 00000000000000000000000000000000 P 9a3648e904f94061bc160bfd15e91a51 [CANDIDATE]: Term 2 election: Requested vote from peers I0507 06:32:35.245726 11187 raft_consensus.cc:2640] T 00000000000000000000000000000000 P 9a3648e904f94061bc160bfd15e91a51 [term 2 FOLLOWER]: Leader election won for term 2 I0507 06:32:35.246784 11187 raft_consensus.cc:667] T 00000000000000000000000000000000 P 9a3648e904f94061bc160bfd15e91a51 [term 2 LEADER]: Becoming Leader. State: Replica: 9a3648e904f94061bc160bfd15e91a51, State: Running, Role: LEADER I0507 06:32:35.247213 11187 consensus_queue.cc:206] T 00000000000000000000000000000000 P 9a3648e904f94061bc160bfd15e91a51 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a3648e904f94061bc160bfd15e91a51" member_type: VOTER } I0507 06:32:35.249310 11189 sys_catalog.cc:337] T 00000000000000000000000000000000 P 9a3648e904f94061bc160bfd15e91a51 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9a3648e904f94061bc160bfd15e91a51. Latest consensus state: current_term: 2 leader_uuid: "9a3648e904f94061bc160bfd15e91a51" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a3648e904f94061bc160bfd15e91a51" member_type: VOTER } } I0507 06:32:35.249676 11189 sys_catalog.cc:340] T 00000000000000000000000000000000 P 9a3648e904f94061bc160bfd15e91a51 [sys.catalog]: This master's current role is: LEADER I0507 06:32:35.249445 11184 sys_catalog.cc:337] T 00000000000000000000000000000000 P 9a3648e904f94061bc160bfd15e91a51 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 2 leader_uuid: "9a3648e904f94061bc160bfd15e91a51" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a3648e904f94061bc160bfd15e91a51" member_type: VOTER } } I0507 06:32:35.250047 11184 sys_catalog.cc:340] T 00000000000000000000000000000000 P 9a3648e904f94061bc160bfd15e91a51 [sys.catalog]: This master's current role is: LEADER I0507 06:32:35.249507 11188 sys_catalog.cc:337] T 00000000000000000000000000000000 P 9a3648e904f94061bc160bfd15e91a51 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "9a3648e904f94061bc160bfd15e91a51" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a3648e904f94061bc160bfd15e91a51" member_type: VOTER } } I0507 06:32:35.250408 11188 sys_catalog.cc:340] T 00000000000000000000000000000000 P 9a3648e904f94061bc160bfd15e91a51 [sys.catalog]: This master's current role is: LEADER I0507 06:32:35.251672 11184 sys_catalog.cc:424] T 00000000000000000000000000000000 P 9a3648e904f94061bc160bfd15e91a51 [sys.catalog]: configured and running, proceeding with master startup. I0507 06:32:35.252089 11191 catalog_manager.cc:1055] Loading table and tablet metadata into memory... I0507 06:32:35.256773 11191 catalog_manager.cc:356] Loaded metadata for table TestTable [id=45840a89cf7145a0bc237c16c9cf54ee] I0507 06:32:35.258487 11191 catalog_manager.cc:410] Loaded metadata for tablet 48ead67853264e088ba5bb88dcf815b0 (table TestTable [id=45840a89cf7145a0bc237c16c9cf54ee]) I0507 06:32:35.258684 11191 catalog_manager.cc:1066] Initializing Kudu internal certificate authority... I0507 06:32:35.261694 11191 catalog_manager.cc:1075] Loading token signing keys... I0507 06:32:35.262781 11191 catalog_manager.cc:4303] T 00000000000000000000000000000000 P 9a3648e904f94061bc160bfd15e91a51: Loaded TSK: 0 WARNING: Logging before InitGoogleLogging() is written to STDERR W0507 06:32:35.408447 11185 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0507 06:32:35.408829 11185 flags.cc:404] Enabled unsafe flag: --never_fsync=true W0507 06:32:35.412029 11185 flags.cc:404] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false W0507 06:32:35.412117 11185 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W0507 06:32:35.412292 11185 flags.cc:404] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.57.193 I0507 06:32:35.419126 11185 tablet_server_main.cc:78] Tablet server non-default flags: --raft_prepare_replacement_before_eviction=false --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_1.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_1.1557210500371863-231/raft_consensus-itest-cluster/ts-0/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.0.57.193:42391 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_1.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data/info.pb --webserver_interface=127.0.57.193 --webserver_port=41695 --heartbeat_incompatible_replica_management_is_fatal=false --heartbeat_interval_ms=500 --tserver_master_addrs=127.0.57.254:39017 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.0.57.193 --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_1.1557210500371863-231/raft_consensus-itest-cluster/ts-0/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.10.0-cdh6.x-SNAPSHOT revision 7d0ee26697926e87af9ef395008799577f6ab43f build type FASTDEBUG built by jenkins at 06 May 2019 22:44:21 PST on quasar-knmpfx-1.vpc.cloudera.com build id 322017 ASAN enabled I0507 06:32:35.430764 11185 system_ntp.cc:202] NTP initialized. Skew: 500ppm Current error: 152016us I0507 06:32:35.431592 11185 fs_manager.cc:263] Metadata directory not provided I0507 06:32:35.431692 11185 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0507 06:32:35.442454 11185 fs_manager.cc:399] Time spent opening directory manager: real 0.009s user 0.004s sys 0.004s I0507 06:32:35.443292 11185 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0507 06:32:35.444030 11185 file_cache.cc:466] Constructed file cache lbm with capacity 419430 I0507 06:32:35.450924 11185 fs_manager.cc:419] Time spent opening block manager: real 0.005s user 0.004s sys 0.000s I0507 06:32:35.451059 11185 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_1.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_1.1557210500371863-231/raft_consensus-itest-cluster/ts-0/wal uuid: "b107758704444e01b3369a5f13ccff50" format_stamp: "Formatted at 2019-05-07 06:32:34 on dist-test-slave-dist-test-slave-rqp6" I0507 06:32:35.452437 11185 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_1.1557210500371863-231/raft_consensus-itest-cluster/ts-0/wal metadata directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_1.1557210500371863-231/raft_consensus-itest-cluster/ts-0/wal 1 data directories: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_1.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0507 06:32:35.493242 11185 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0507 06:32:35.495460 11185 ts_tablet_manager.cc:344] Loading tablet metadata (0/1 complete) I0507 06:32:35.499770 11185 ts_tablet_manager.cc:356] Loaded tablet metadata (1 total tablets, 1 live tablets) I0507 06:32:35.499832 11185 ts_tablet_manager.cc:362] Registering tablets (0/1 complete) I0507 06:32:35.504885 11185 ts_tablet_manager.cc:376] Registered 1 tablets I0507 06:32:35.505405 11209 tablet_bootstrap.cc:439] T 48ead67853264e088ba5bb88dcf815b0 P b107758704444e01b3369a5f13ccff50: Bootstrap starting. I0507 06:32:35.520942 11209 log.cc:525] T 48ead67853264e088ba5bb88dcf815b0 P b107758704444e01b3369a5f13ccff50: Log is configured to *not* fsync() on all Append() calls I0507 06:32:35.528218 11209 tablet_bootstrap.cc:439] T 48ead67853264e088ba5bb88dcf815b0 P b107758704444e01b3369a5f13ccff50: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I0507 06:32:35.529008 11209 tablet_bootstrap.cc:439] T 48ead67853264e088ba5bb88dcf815b0 P b107758704444e01b3369a5f13ccff50: Bootstrap complete. I0507 06:32:35.530305 11209 ts_tablet_manager.cc:1053] T 48ead67853264e088ba5bb88dcf815b0 P b107758704444e01b3369a5f13ccff50: Time spent bootstrapping tablet: real 0.025s user 0.013s sys 0.010s I0507 06:32:35.546196 11209 raft_consensus.cc:340] T 48ead67853264e088ba5bb88dcf815b0 P b107758704444e01b3369a5f13ccff50 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b107758704444e01b3369a5f13ccff50" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 42391 } } I0507 06:32:35.549124 11209 raft_consensus.cc:705] T 48ead67853264e088ba5bb88dcf815b0 P b107758704444e01b3369a5f13ccff50 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b107758704444e01b3369a5f13ccff50, State: Initialized, Role: FOLLOWER I0507 06:32:35.549784 11209 consensus_queue.cc:229] T 48ead67853264e088ba5bb88dcf815b0 P b107758704444e01b3369a5f13ccff50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b107758704444e01b3369a5f13ccff50" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 42391 } } I0507 06:32:35.549991 11209 raft_consensus.cc:380] T 48ead67853264e088ba5bb88dcf815b0 P b107758704444e01b3369a5f13ccff50 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0507 06:32:35.550307 11209 raft_consensus.cc:472] T 48ead67853264e088ba5bb88dcf815b0 P b107758704444e01b3369a5f13ccff50 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0507 06:32:35.550451 11209 raft_consensus.cc:2886] T 48ead67853264e088ba5bb88dcf815b0 P b107758704444e01b3369a5f13ccff50 [term 1 FOLLOWER]: Advancing to term 2 I0507 06:32:35.553901 11209 raft_consensus.cc:494] T 48ead67853264e088ba5bb88dcf815b0 P b107758704444e01b3369a5f13ccff50 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b107758704444e01b3369a5f13ccff50" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 42391 } } I0507 06:32:35.554374 11209 leader_election.cc:310] T 48ead67853264e088ba5bb88dcf815b0 P b107758704444e01b3369a5f13ccff50 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b107758704444e01b3369a5f13ccff50; no voters: I0507 06:32:35.555284 11209 leader_election.cc:296] T 48ead67853264e088ba5bb88dcf815b0 P b107758704444e01b3369a5f13ccff50 [CANDIDATE]: Term 2 election: Requested vote from peers I0507 06:32:35.555506 11241 raft_consensus.cc:2640] T 48ead67853264e088ba5bb88dcf815b0 P b107758704444e01b3369a5f13ccff50 [term 2 FOLLOWER]: Leader election won for term 2 I0507 06:32:35.557365 11241 raft_consensus.cc:667] T 48ead67853264e088ba5bb88dcf815b0 P b107758704444e01b3369a5f13ccff50 [term 2 LEADER]: Becoming Leader. State: Replica: b107758704444e01b3369a5f13ccff50, State: Running, Role: LEADER I0507 06:32:35.557685 11209 ts_tablet_manager.cc:1081] T 48ead67853264e088ba5bb88dcf815b0 P b107758704444e01b3369a5f13ccff50: Time spent starting tablet: real 0.027s user 0.025s sys 0.002s I0507 06:32:35.558018 11241 consensus_queue.cc:206] T 48ead67853264e088ba5bb88dcf815b0 P b107758704444e01b3369a5f13ccff50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b107758704444e01b3369a5f13ccff50" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 42391 } } I0507 06:32:35.645879 11185 rpc_server.cc:205] RPC server started. Bound to: 127.0.57.193:42391 I0507 06:32:35.648806 11185 webserver.cc:308] Webserver started at http://127.0.57.193:41695/ using document root and password file I0507 06:32:35.649559 11185 server_base.cc:608] Dumped server information to /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_1.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data/info.pb I0507 06:32:35.652772 231 external_mini_cluster.cc:995] Started /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver as pid 11185 I0507 06:32:35.663558 11320 heartbeater.cc:345] Connected to a master server at 127.0.57.254:39017 I0507 06:32:35.663766 11320 heartbeater.cc:426] Registering TS with master... I0507 06:32:35.664229 11320 heartbeater.cc:475] Master 127.0.57.254:39017 requested a full tablet report, sending... W0507 06:32:35.666219 11151 master_service.cc:205] Configuration error: replica replacement scheme (EVICT_FIRST) of the tablet server b107758704444e01b3369a5f13ccff50 at 127.0.57.193:37387 differs from the catalog manager's (PREPARE_REPLACEMENT_BEFORE_EVICTION); they must be run with the same scheme (controlled by the --raft_prepare_replacement_before_eviction flag) W0507 06:32:35.667197 11320 heartbeater.cc:587] Failed to heartbeat to 127.0.57.254:39017 (0 consecutive failures): Configuration error: replica replacement scheme (EVICT_FIRST) of the tablet server b107758704444e01b3369a5f13ccff50 at 127.0.57.193:37387 differs from the catalog manager's (PREPARE_REPLACEMENT_BEFORE_EVICTION); they must be run with the same scheme (controlled by the --raft_prepare_replacement_before_eviction flag) E0507 06:32:35.667318 11320 heartbeater.cc:622] master detected incompatibility: Configuration error: replica replacement scheme (EVICT_FIRST) of the tablet server b107758704444e01b3369a5f13ccff50 at 127.0.57.193:37387 differs from the catalog manager's (PREPARE_REPLACEMENT_BEFORE_EVICTION); they must be run with the same scheme (controlled by the --raft_prepare_replacement_before_eviction flag) I0507 06:32:35.667433 11320 heartbeater.cc:426] Registering TS with master... I0507 06:32:35.667634 11320 heartbeater.cc:475] Master 127.0.57.254:39017 requested a full tablet report, sending... W0507 06:32:35.668404 11151 master_service.cc:205] Configuration error: replica replacement scheme (EVICT_FIRST) of the tablet server b107758704444e01b3369a5f13ccff50 at 127.0.57.193:37387 differs from the catalog manager's (PREPARE_REPLACEMENT_BEFORE_EVICTION); they must be run with the same scheme (controlled by the --raft_prepare_replacement_before_eviction flag) I0507 06:32:35.668890 11320 heartbeater.cc:426] Registering TS with master... I0507 06:32:35.669066 11320 heartbeater.cc:475] Master 127.0.57.254:39017 requested a full tablet report, sending... W0507 06:32:35.669749 11151 master_service.cc:205] Configuration error: replica replacement scheme (EVICT_FIRST) of the tablet server b107758704444e01b3369a5f13ccff50 at 127.0.57.193:37387 differs from the catalog manager's (PREPARE_REPLACEMENT_BEFORE_EVICTION); they must be run with the same scheme (controlled by the --raft_prepare_replacement_before_eviction flag) W0507 06:32:35.670198 11320 heartbeater.cc:380] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts. I0507 06:32:36.170454 11320 heartbeater.cc:426] Registering TS with master... I0507 06:32:36.170848 11320 heartbeater.cc:475] Master 127.0.57.254:39017 requested a full tablet report, sending... W0507 06:32:36.172221 11151 master_service.cc:205] Configuration error: replica replacement scheme (EVICT_FIRST) of the tablet server b107758704444e01b3369a5f13ccff50 at 127.0.57.193:37387 differs from the catalog manager's (PREPARE_REPLACEMENT_BEFORE_EVICTION); they must be run with the same scheme (controlled by the --raft_prepare_replacement_before_eviction flag) I0507 06:32:36.673136 11320 heartbeater.cc:426] Registering TS with master... I0507 06:32:36.673501 11320 heartbeater.cc:475] Master 127.0.57.254:39017 requested a full tablet report, sending... W0507 06:32:36.675137 11151 master_service.cc:205] Configuration error: replica replacement scheme (EVICT_FIRST) of the tablet server b107758704444e01b3369a5f13ccff50 at 127.0.57.193:37387 differs from the catalog manager's (PREPARE_REPLACEMENT_BEFORE_EVICTION); they must be run with the same scheme (controlled by the --raft_prepare_replacement_before_eviction flag) I0507 06:32:37.159096 231 external_mini_cluster.cc:1172] Attempting to check leaks for /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver pid 11185 I0507 06:32:37.165410 11317 generic_service.cc:171] Checking for leaks (request via RPC) I0507 06:32:37.209664 11320 heartbeater.cc:426] Registering TS with master... I0507 06:32:37.210007 11320 heartbeater.cc:475] Master 127.0.57.254:39017 requested a full tablet report, sending... I0507 06:32:37.211113 231 external_mini_cluster.cc:1131] Killing /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver with pid 11185 W0507 06:32:37.211457 11151 master_service.cc:205] Configuration error: replica replacement scheme (EVICT_FIRST) of the tablet server b107758704444e01b3369a5f13ccff50 at 127.0.57.193:37387 differs from the catalog manager's (PREPARE_REPLACEMENT_BEFORE_EVICTION); they must be run with the same scheme (controlled by the --raft_prepare_replacement_before_eviction flag) W0507 06:32:37.223095 11132 connection.cc:647] server connection from 127.0.57.193:37387 recv error: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107) I0507 06:32:37.223542 231 external_mini_cluster.cc:933] Running /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_1.1557210500371863-231/raft_consensus-itest-cluster/ts-0/wal --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_1.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data --block_manager=log --rpc_bind_addresses=127.0.57.193:42391 --local_ip_for_outbound_sockets=127.0.57.193 --webserver_port=41695 --webserver_interface=127.0.57.193 --tserver_master_addrs=127.0.57.254:39017 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_1.1557210500371863-231/raft_consensus-itest-cluster/ts-0/logs --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_1.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --heartbeat_incompatible_replica_management_is_fatal=false --heartbeat_interval_ms=500 --raft_prepare_replacement_before_eviction=true --heartbeat_inject_required_feature_flag=999 with env {} WARNING: Logging before InitGoogleLogging() is written to STDERR W0507 06:32:37.396149 11328 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0507 06:32:37.396384 11328 flags.cc:404] Enabled unsafe flag: --heartbeat_inject_required_feature_flag=999 W0507 06:32:37.396447 11328 flags.cc:404] Enabled unsafe flag: --never_fsync=true W0507 06:32:37.398586 11328 flags.cc:404] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true W0507 06:32:37.398675 11328 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W0507 06:32:37.398818 11328 flags.cc:404] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.57.193 I0507 06:32:37.402846 11328 tablet_server_main.cc:78] Tablet server non-default flags: --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_1.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_1.1557210500371863-231/raft_consensus-itest-cluster/ts-0/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.0.57.193:42391 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_1.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data/info.pb --webserver_interface=127.0.57.193 --webserver_port=41695 --heartbeat_incompatible_replica_management_is_fatal=false --heartbeat_inject_required_feature_flag=999 --heartbeat_interval_ms=500 --tserver_master_addrs=127.0.57.254:39017 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.0.57.193 --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_1.1557210500371863-231/raft_consensus-itest-cluster/ts-0/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.10.0-cdh6.x-SNAPSHOT revision 7d0ee26697926e87af9ef395008799577f6ab43f build type FASTDEBUG built by jenkins at 06 May 2019 22:44:21 PST on quasar-knmpfx-1.vpc.cloudera.com build id 322017 ASAN enabled I0507 06:32:37.410382 11328 system_ntp.cc:202] NTP initialized. Skew: 500ppm Current error: 153016us I0507 06:32:37.411008 11328 fs_manager.cc:263] Metadata directory not provided I0507 06:32:37.411082 11328 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0507 06:32:37.416514 11328 fs_manager.cc:399] Time spent opening directory manager: real 0.004s user 0.001s sys 0.001s I0507 06:32:37.417152 11328 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0507 06:32:37.417687 11328 file_cache.cc:466] Constructed file cache lbm with capacity 419430 I0507 06:32:37.421258 11328 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s I0507 06:32:37.421340 11328 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_1.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_1.1557210500371863-231/raft_consensus-itest-cluster/ts-0/wal uuid: "b107758704444e01b3369a5f13ccff50" format_stamp: "Formatted at 2019-05-07 06:32:34 on dist-test-slave-dist-test-slave-rqp6" I0507 06:32:37.422334 11328 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_1.1557210500371863-231/raft_consensus-itest-cluster/ts-0/wal metadata directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_1.1557210500371863-231/raft_consensus-itest-cluster/ts-0/wal 1 data directories: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_1.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0507 06:32:37.447355 11328 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0507 06:32:37.449107 11328 ts_tablet_manager.cc:344] Loading tablet metadata (0/1 complete) I0507 06:32:37.452915 11328 ts_tablet_manager.cc:356] Loaded tablet metadata (1 total tablets, 1 live tablets) I0507 06:32:37.452988 11328 ts_tablet_manager.cc:362] Registering tablets (0/1 complete) I0507 06:32:37.455463 11328 ts_tablet_manager.cc:376] Registered 1 tablets I0507 06:32:37.455920 11343 tablet_bootstrap.cc:439] T 48ead67853264e088ba5bb88dcf815b0 P b107758704444e01b3369a5f13ccff50: Bootstrap starting. I0507 06:32:37.468403 11343 log.cc:525] T 48ead67853264e088ba5bb88dcf815b0 P b107758704444e01b3369a5f13ccff50: Log is configured to *not* fsync() on all Append() calls I0507 06:32:37.472365 11343 tablet_bootstrap.cc:439] T 48ead67853264e088ba5bb88dcf815b0 P b107758704444e01b3369a5f13ccff50: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I0507 06:32:37.472903 11343 tablet_bootstrap.cc:439] T 48ead67853264e088ba5bb88dcf815b0 P b107758704444e01b3369a5f13ccff50: Bootstrap complete. I0507 06:32:37.473847 11343 ts_tablet_manager.cc:1053] T 48ead67853264e088ba5bb88dcf815b0 P b107758704444e01b3369a5f13ccff50: Time spent bootstrapping tablet: real 0.018s user 0.007s sys 0.009s I0507 06:32:37.482970 11343 raft_consensus.cc:340] T 48ead67853264e088ba5bb88dcf815b0 P b107758704444e01b3369a5f13ccff50 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b107758704444e01b3369a5f13ccff50" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 42391 } } I0507 06:32:37.486582 11343 raft_consensus.cc:705] T 48ead67853264e088ba5bb88dcf815b0 P b107758704444e01b3369a5f13ccff50 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: b107758704444e01b3369a5f13ccff50, State: Initialized, Role: FOLLOWER I0507 06:32:37.487257 11343 consensus_queue.cc:229] T 48ead67853264e088ba5bb88dcf815b0 P b107758704444e01b3369a5f13ccff50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b107758704444e01b3369a5f13ccff50" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 42391 } } I0507 06:32:37.487812 11343 raft_consensus.cc:380] T 48ead67853264e088ba5bb88dcf815b0 P b107758704444e01b3369a5f13ccff50 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0507 06:32:37.487979 11343 raft_consensus.cc:472] T 48ead67853264e088ba5bb88dcf815b0 P b107758704444e01b3369a5f13ccff50 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0507 06:32:37.488265 11343 raft_consensus.cc:2886] T 48ead67853264e088ba5bb88dcf815b0 P b107758704444e01b3369a5f13ccff50 [term 2 FOLLOWER]: Advancing to term 3 I0507 06:32:37.492373 11343 raft_consensus.cc:494] T 48ead67853264e088ba5bb88dcf815b0 P b107758704444e01b3369a5f13ccff50 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b107758704444e01b3369a5f13ccff50" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 42391 } } I0507 06:32:37.493116 11343 leader_election.cc:310] T 48ead67853264e088ba5bb88dcf815b0 P b107758704444e01b3369a5f13ccff50 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b107758704444e01b3369a5f13ccff50; no voters: I0507 06:32:37.494081 11343 leader_election.cc:296] T 48ead67853264e088ba5bb88dcf815b0 P b107758704444e01b3369a5f13ccff50 [CANDIDATE]: Term 3 election: Requested vote from peers I0507 06:32:37.494455 11370 raft_consensus.cc:2640] T 48ead67853264e088ba5bb88dcf815b0 P b107758704444e01b3369a5f13ccff50 [term 3 FOLLOWER]: Leader election won for term 3 I0507 06:32:37.495573 11343 ts_tablet_manager.cc:1081] T 48ead67853264e088ba5bb88dcf815b0 P b107758704444e01b3369a5f13ccff50: Time spent starting tablet: real 0.022s user 0.008s sys 0.013s I0507 06:32:37.495944 11370 raft_consensus.cc:667] T 48ead67853264e088ba5bb88dcf815b0 P b107758704444e01b3369a5f13ccff50 [term 3 LEADER]: Becoming Leader. State: Replica: b107758704444e01b3369a5f13ccff50, State: Running, Role: LEADER I0507 06:32:37.496660 11370 consensus_queue.cc:206] T 48ead67853264e088ba5bb88dcf815b0 P b107758704444e01b3369a5f13ccff50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b107758704444e01b3369a5f13ccff50" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 42391 } } I0507 06:32:37.513691 11328 rpc_server.cc:205] RPC server started. Bound to: 127.0.57.193:42391 I0507 06:32:37.514803 11328 webserver.cc:308] Webserver started at http://127.0.57.193:41695/ using document root and password file I0507 06:32:37.515442 11328 server_base.cc:608] Dumped server information to /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_1.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data/info.pb I0507 06:32:37.525871 231 external_mini_cluster.cc:995] Started /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver as pid 11328 I0507 06:32:37.530320 11454 heartbeater.cc:345] Connected to a master server at 127.0.57.254:39017 I0507 06:32:37.530571 11454 heartbeater.cc:426] Registering TS with master... I0507 06:32:37.531245 11454 heartbeater.cc:475] Master 127.0.57.254:39017 requested a full tablet report, sending... W0507 06:32:37.532925 11133 messenger.cc:389] Unable to handle RPC call: Not implemented: call requires unsupported application feature flags: 999 W0507 06:32:37.533546 11454 heartbeater.cc:587] Failed to heartbeat to 127.0.57.254:39017 (0 consecutive failures): Remote error: Failed to send heartbeat to master: unsupported feature flags E0507 06:32:37.533689 11454 heartbeater.cc:622] master does not support required feature flags: Remote error: Failed to send heartbeat to master: unsupported feature flags I0507 06:32:37.533766 11454 heartbeater.cc:426] Registering TS with master... I0507 06:32:37.533967 11454 heartbeater.cc:475] Master 127.0.57.254:39017 requested a full tablet report, sending... W0507 06:32:37.534596 11133 messenger.cc:389] Unable to handle RPC call: Not implemented: call requires unsupported application feature flags: 999 I0507 06:32:37.534972 11454 heartbeater.cc:426] Registering TS with master... I0507 06:32:37.535130 11454 heartbeater.cc:475] Master 127.0.57.254:39017 requested a full tablet report, sending... W0507 06:32:37.535703 11133 messenger.cc:389] Unable to handle RPC call: Not implemented: call requires unsupported application feature flags: 999 W0507 06:32:37.536039 11454 heartbeater.cc:380] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts. I0507 06:32:38.036227 11454 heartbeater.cc:426] Registering TS with master... I0507 06:32:38.036631 11454 heartbeater.cc:475] Master 127.0.57.254:39017 requested a full tablet report, sending... W0507 06:32:38.037976 11133 messenger.cc:389] Unable to handle RPC call: Not implemented: call requires unsupported application feature flags: 999 I0507 06:32:38.538911 11454 heartbeater.cc:426] Registering TS with master... I0507 06:32:38.539458 11454 heartbeater.cc:475] Master 127.0.57.254:39017 requested a full tablet report, sending... W0507 06:32:38.540762 11133 messenger.cc:389] Unable to handle RPC call: Not implemented: call requires unsupported application feature flags: 999 I0507 06:32:39.028769 231 external_mini_cluster.cc:1172] Attempting to check leaks for /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver pid 11328 I0507 06:32:39.036772 11451 generic_service.cc:171] Checking for leaks (request via RPC) I0507 06:32:39.081413 11454 heartbeater.cc:426] Registering TS with master... I0507 06:32:39.081845 11454 heartbeater.cc:475] Master 127.0.57.254:39017 requested a full tablet report, sending... I0507 06:32:39.082763 231 external_mini_cluster.cc:1131] Killing /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver with pid 11328 W0507 06:32:39.083061 11133 messenger.cc:389] Unable to handle RPC call: Not implemented: call requires unsupported application feature flags: 999 W0507 06:32:39.092648 11133 connection.cc:647] server connection from 127.0.57.193:35431 recv error: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107) I0507 06:32:39.092826 231 external_mini_cluster.cc:1172] Attempting to check leaks for /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-master pid 11125 I0507 06:32:39.093839 11181 generic_service.cc:171] Checking for leaks (request via RPC) I0507 06:32:39.163645 231 external_mini_cluster.cc:1131] Killing /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-master with pid 11125 [ OK ] IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig/1 (5208 ms) [ RUN ] IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig/2 I0507 06:32:39.185305 231 test_util.cc:212] Using random seed: 1646543750 I0507 06:32:39.186843 231 ts_itest-base.cc:112] Starting cluster with: I0507 06:32:39.186879 231 ts_itest-base.cc:113] -------------- I0507 06:32:39.186906 231 ts_itest-base.cc:114] 1 tablet servers I0507 06:32:39.186930 231 ts_itest-base.cc:115] 1 replicas per TS I0507 06:32:39.186955 231 ts_itest-base.cc:116] -------------- I0507 06:32:39.188127 231 external_mini_cluster.cc:933] Running /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-master /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-master --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_2.1557210500371863-231/raft_consensus-itest-cluster/master-0/wal --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_2.1557210500371863-231/raft_consensus-itest-cluster/master-0/data --block_manager=log --webserver_interface=localhost --ipki_ca_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.0.57.254:42725 --webserver_interface=127.0.57.254 --webserver_port=0 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_2.1557210500371863-231/raft_consensus-itest-cluster/master-0/logs --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_2.1557210500371863-231/raft_consensus-itest-cluster/master-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=false --rpc_reuseport=true with env {} WARNING: Logging before InitGoogleLogging() is written to STDERR W0507 06:32:39.399027 11466 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0507 06:32:39.399274 11466 flags.cc:404] Enabled unsafe flag: --never_fsync=true W0507 06:32:39.401782 11466 flags.cc:404] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false W0507 06:32:39.401846 11466 flags.cc:404] Enabled experimental flag: --ipki_ca_key_size=1024 W0507 06:32:39.401880 11466 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W0507 06:32:39.401902 11466 flags.cc:404] Enabled experimental flag: --tsk_num_rsa_bits=512 W0507 06:32:39.401957 11466 flags.cc:404] Enabled experimental flag: --rpc_reuseport=true I0507 06:32:39.407369 11466 master_main.cc:73] Master server non-default flags: --raft_prepare_replacement_before_eviction=false --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_2.1557210500371863-231/raft_consensus-itest-cluster/master-0/data --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_2.1557210500371863-231/raft_consensus-itest-cluster/master-0/wal --ipki_ca_key_size=1024 --ipki_server_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.0.57.254:42725 --rpc_reuseport=true --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_2.1557210500371863-231/raft_consensus-itest-cluster/master-0/data/info.pb --webserver_interface=127.0.57.254 --webserver_port=0 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_2.1557210500371863-231/raft_consensus-itest-cluster/master-0/logs --logbuflevel=-1 --logtostderr=true Master server version: kudu 1.10.0-cdh6.x-SNAPSHOT revision 7d0ee26697926e87af9ef395008799577f6ab43f build type FASTDEBUG built by jenkins at 06 May 2019 22:44:21 PST on quasar-knmpfx-1.vpc.cloudera.com build id 322017 ASAN enabled I0507 06:32:39.414660 11466 system_ntp.cc:202] NTP initialized. Skew: 500ppm Current error: 154016us I0507 06:32:39.416049 11466 fs_manager.cc:263] Metadata directory not provided I0507 06:32:39.416118 11466 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0507 06:32:39.416424 11466 server_base.cc:433] Could not load existing FS layout: Not found: could not find a healthy instance file I0507 06:32:39.416501 11466 server_base.cc:434] Attempting to create new FS layout instead I0507 06:32:39.419217 11466 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_2.1557210500371863-231/raft_consensus-itest-cluster/master-0/data/instance: uuid: "4c6c527837014acf9fc19b31112f1a4a" format_stamp: "Formatted at 2019-05-07 06:32:39 on dist-test-slave-dist-test-slave-rqp6" I0507 06:32:39.419782 11466 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_2.1557210500371863-231/raft_consensus-itest-cluster/master-0/wal/instance: uuid: "4c6c527837014acf9fc19b31112f1a4a" format_stamp: "Formatted at 2019-05-07 06:32:39 on dist-test-slave-dist-test-slave-rqp6" I0507 06:32:39.427145 11466 fs_manager.cc:503] Time spent creating directory manager: real 0.007s user 0.004s sys 0.004s I0507 06:32:39.428268 11466 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0507 06:32:39.428921 11466 file_cache.cc:466] Constructed file cache lbm with capacity 419430 I0507 06:32:39.432176 11466 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s I0507 06:32:39.432267 11466 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_2.1557210500371863-231/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_2.1557210500371863-231/raft_consensus-itest-cluster/master-0/wal uuid: "4c6c527837014acf9fc19b31112f1a4a" format_stamp: "Formatted at 2019-05-07 06:32:39 on dist-test-slave-dist-test-slave-rqp6" I0507 06:32:39.432415 11466 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_2.1557210500371863-231/raft_consensus-itest-cluster/master-0/wal metadata directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_2.1557210500371863-231/raft_consensus-itest-cluster/master-0/wal 1 data directories: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_2.1557210500371863-231/raft_consensus-itest-cluster/master-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0507 06:32:39.468412 11466 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0507 06:32:39.484882 11466 rpc_server.cc:205] RPC server started. Bound to: 127.0.57.254:42725 I0507 06:32:39.486214 11466 webserver.cc:308] Webserver started at http://127.0.57.254:35977/ using document root and password file I0507 06:32:39.487005 11466 server_base.cc:608] Dumped server information to /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_2.1557210500371863-231/raft_consensus-itest-cluster/master-0/data/info.pb I0507 06:32:39.488435 231 external_mini_cluster.cc:995] Started /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-master as pid 11466 I0507 06:32:39.489063 231 external_mini_cluster.cc:933] Running /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_2.1557210500371863-231/raft_consensus-itest-cluster/ts-0/wal --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_2.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data --block_manager=log --rpc_bind_addresses=127.0.57.193:0 --local_ip_for_outbound_sockets=127.0.57.193 --webserver_interface=127.0.57.193 --webserver_port=0 --tserver_master_addrs=127.0.57.254:42725 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_2.1557210500371863-231/raft_consensus-itest-cluster/ts-0/logs --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_2.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --heartbeat_incompatible_replica_management_is_fatal=true --heartbeat_interval_ms=500 --raft_prepare_replacement_before_eviction=false with env {} I0507 06:32:39.490988 11525 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0507 06:32:39.507406 11525 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 4c6c527837014acf9fc19b31112f1a4a: Bootstrap starting. I0507 06:32:39.514866 11525 tablet_bootstrap.cc:590] T 00000000000000000000000000000000 P 4c6c527837014acf9fc19b31112f1a4a: No blocks or log segments found. Creating new log. I0507 06:32:39.515942 11525 log.cc:525] T 00000000000000000000000000000000 P 4c6c527837014acf9fc19b31112f1a4a: Log is configured to *not* fsync() on all Append() calls I0507 06:32:39.526207 11525 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 4c6c527837014acf9fc19b31112f1a4a: No bootstrap required, opened a new log I0507 06:32:39.533301 11525 raft_consensus.cc:340] T 00000000000000000000000000000000 P 4c6c527837014acf9fc19b31112f1a4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c6c527837014acf9fc19b31112f1a4a" member_type: VOTER } I0507 06:32:39.533586 11525 raft_consensus.cc:366] T 00000000000000000000000000000000 P 4c6c527837014acf9fc19b31112f1a4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0507 06:32:39.533643 11525 raft_consensus.cc:705] T 00000000000000000000000000000000 P 4c6c527837014acf9fc19b31112f1a4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c6c527837014acf9fc19b31112f1a4a, State: Initialized, Role: FOLLOWER I0507 06:32:39.534165 11525 consensus_queue.cc:229] T 00000000000000000000000000000000 P 4c6c527837014acf9fc19b31112f1a4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c6c527837014acf9fc19b31112f1a4a" member_type: VOTER } I0507 06:32:39.534345 11525 raft_consensus.cc:380] T 00000000000000000000000000000000 P 4c6c527837014acf9fc19b31112f1a4a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0507 06:32:39.534427 11525 raft_consensus.cc:472] T 00000000000000000000000000000000 P 4c6c527837014acf9fc19b31112f1a4a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0507 06:32:39.534516 11525 raft_consensus.cc:2886] T 00000000000000000000000000000000 P 4c6c527837014acf9fc19b31112f1a4a [term 0 FOLLOWER]: Advancing to term 1 I0507 06:32:39.536188 11525 raft_consensus.cc:494] T 00000000000000000000000000000000 P 4c6c527837014acf9fc19b31112f1a4a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c6c527837014acf9fc19b31112f1a4a" member_type: VOTER } I0507 06:32:39.536686 11525 leader_election.cc:310] T 00000000000000000000000000000000 P 4c6c527837014acf9fc19b31112f1a4a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4c6c527837014acf9fc19b31112f1a4a; no voters: I0507 06:32:39.537497 11525 leader_election.cc:296] T 00000000000000000000000000000000 P 4c6c527837014acf9fc19b31112f1a4a [CANDIDATE]: Term 1 election: Requested vote from peers I0507 06:32:39.537755 11528 raft_consensus.cc:2640] T 00000000000000000000000000000000 P 4c6c527837014acf9fc19b31112f1a4a [term 1 FOLLOWER]: Leader election won for term 1 I0507 06:32:39.538868 11528 raft_consensus.cc:667] T 00000000000000000000000000000000 P 4c6c527837014acf9fc19b31112f1a4a [term 1 LEADER]: Becoming Leader. State: Replica: 4c6c527837014acf9fc19b31112f1a4a, State: Running, Role: LEADER I0507 06:32:39.539363 11528 consensus_queue.cc:206] T 00000000000000000000000000000000 P 4c6c527837014acf9fc19b31112f1a4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c6c527837014acf9fc19b31112f1a4a" member_type: VOTER } I0507 06:32:39.543020 11529 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4c6c527837014acf9fc19b31112f1a4a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4c6c527837014acf9fc19b31112f1a4a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c6c527837014acf9fc19b31112f1a4a" member_type: VOTER } } I0507 06:32:39.543299 11529 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4c6c527837014acf9fc19b31112f1a4a [sys.catalog]: This master's current role is: LEADER I0507 06:32:39.544711 11530 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4c6c527837014acf9fc19b31112f1a4a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4c6c527837014acf9fc19b31112f1a4a. Latest consensus state: current_term: 1 leader_uuid: "4c6c527837014acf9fc19b31112f1a4a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c6c527837014acf9fc19b31112f1a4a" member_type: VOTER } } I0507 06:32:39.545011 11530 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4c6c527837014acf9fc19b31112f1a4a [sys.catalog]: This master's current role is: LEADER I0507 06:32:39.545020 11525 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4c6c527837014acf9fc19b31112f1a4a [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 1 leader_uuid: "4c6c527837014acf9fc19b31112f1a4a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c6c527837014acf9fc19b31112f1a4a" member_type: VOTER } } I0507 06:32:39.545125 11525 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4c6c527837014acf9fc19b31112f1a4a [sys.catalog]: This master's current role is: LEADER I0507 06:32:39.546404 11525 sys_catalog.cc:424] T 00000000000000000000000000000000 P 4c6c527837014acf9fc19b31112f1a4a [sys.catalog]: configured and running, proceeding with master startup. I0507 06:32:39.546600 11534 catalog_manager.cc:1055] Loading table and tablet metadata into memory... I0507 06:32:39.554859 11534 catalog_manager.cc:1066] Initializing Kudu internal certificate authority... I0507 06:32:39.654505 11534 catalog_manager.cc:950] Generated new certificate authority record I0507 06:32:39.655900 11534 catalog_manager.cc:1075] Loading token signing keys... I0507 06:32:39.661975 11534 catalog_manager.cc:4292] T 00000000000000000000000000000000 P 4c6c527837014acf9fc19b31112f1a4a: Generated new TSK 0 WARNING: Logging before InitGoogleLogging() is written to STDERR W0507 06:32:39.676941 11526 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0507 06:32:39.677165 11526 flags.cc:404] Enabled unsafe flag: --never_fsync=true W0507 06:32:39.679328 11526 flags.cc:404] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false W0507 06:32:39.679399 11526 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W0507 06:32:39.679497 11526 flags.cc:404] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.57.193 I0507 06:32:39.683490 11526 tablet_server_main.cc:78] Tablet server non-default flags: --raft_prepare_replacement_before_eviction=false --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_2.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_2.1557210500371863-231/raft_consensus-itest-cluster/ts-0/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.0.57.193:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_2.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data/info.pb --webserver_interface=127.0.57.193 --webserver_port=0 --heartbeat_interval_ms=500 --tserver_master_addrs=127.0.57.254:42725 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.0.57.193 --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_2.1557210500371863-231/raft_consensus-itest-cluster/ts-0/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.10.0-cdh6.x-SNAPSHOT revision 7d0ee26697926e87af9ef395008799577f6ab43f build type FASTDEBUG built by jenkins at 06 May 2019 22:44:21 PST on quasar-knmpfx-1.vpc.cloudera.com build id 322017 ASAN enabled I0507 06:32:39.691282 11526 system_ntp.cc:202] NTP initialized. Skew: 500ppm Current error: 154016us I0507 06:32:39.691962 11526 fs_manager.cc:263] Metadata directory not provided I0507 06:32:39.692025 11526 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0507 06:32:39.692323 11526 server_base.cc:433] Could not load existing FS layout: Not found: could not find a healthy instance file I0507 06:32:39.692399 11526 server_base.cc:434] Attempting to create new FS layout instead I0507 06:32:39.695029 11526 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_2.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data/instance: uuid: "fa8f2d28bec94836a74c64cfb9ce6a54" format_stamp: "Formatted at 2019-05-07 06:32:39 on dist-test-slave-dist-test-slave-rqp6" I0507 06:32:39.695606 11526 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_2.1557210500371863-231/raft_consensus-itest-cluster/ts-0/wal/instance: uuid: "fa8f2d28bec94836a74c64cfb9ce6a54" format_stamp: "Formatted at 2019-05-07 06:32:39 on dist-test-slave-dist-test-slave-rqp6" I0507 06:32:39.702325 11526 fs_manager.cc:503] Time spent creating directory manager: real 0.007s user 0.005s sys 0.001s I0507 06:32:39.703379 11526 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0507 06:32:39.703933 11526 file_cache.cc:466] Constructed file cache lbm with capacity 419430 I0507 06:32:39.706945 11526 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.000s sys 0.000s I0507 06:32:39.707034 11526 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_2.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_2.1557210500371863-231/raft_consensus-itest-cluster/ts-0/wal uuid: "fa8f2d28bec94836a74c64cfb9ce6a54" format_stamp: "Formatted at 2019-05-07 06:32:39 on dist-test-slave-dist-test-slave-rqp6" I0507 06:32:39.707183 11526 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_2.1557210500371863-231/raft_consensus-itest-cluster/ts-0/wal metadata directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_2.1557210500371863-231/raft_consensus-itest-cluster/ts-0/wal 1 data directories: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_2.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0507 06:32:39.726186 11526 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0507 06:32:39.727915 11526 ts_tablet_manager.cc:356] Loaded tablet metadata (0 total tablets, 0 live tablets) I0507 06:32:39.727994 11526 ts_tablet_manager.cc:376] Registered 0 tablets I0507 06:32:39.775688 11526 rpc_server.cc:205] RPC server started. Bound to: 127.0.57.193:36629 I0507 06:32:39.777184 11526 webserver.cc:308] Webserver started at http://127.0.57.193:43791/ using document root and password file I0507 06:32:39.778009 11526 server_base.cc:608] Dumped server information to /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_2.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data/info.pb I0507 06:32:39.782253 231 external_mini_cluster.cc:995] Started /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver as pid 11526 I0507 06:32:39.795599 11656 heartbeater.cc:345] Connected to a master server at 127.0.57.254:42725 I0507 06:32:39.795836 11656 heartbeater.cc:426] Registering TS with master... I0507 06:32:39.796337 11656 heartbeater.cc:475] Master 127.0.57.254:42725 requested a full tablet report, sending... I0507 06:32:39.798137 11492 ts_manager.cc:159] Registered new tserver with Master: fa8f2d28bec94836a74c64cfb9ce6a54 (127.0.57.193:36629) I0507 06:32:39.799998 11492 master_service.cc:273] Signed X509 certificate for tserver {username='slave'} at 127.0.57.193:33951 I0507 06:32:39.800120 231 external_mini_cluster.cc:623] 1 TS(s) registered with all masters I0507 06:32:39.814934 11491 catalog_manager.cc:1415] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33098: name: "TestTable" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 1 split_rows_range_bounds { } partition_schema { range_schema { columns { name: "key" } } } I0507 06:32:39.845553 11593 tablet_service.cc:1016] Processing CreateTablet for tablet 826e8360d3e74edf8ea66f29380738ce (table=TestTable [id=ad86553e031344aebd031dbb7c8b7bae]), partition=RANGE (key) PARTITION UNBOUNDED I0507 06:32:39.846554 11593 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet 826e8360d3e74edf8ea66f29380738ce. 1 dirs total, 0 dirs full, 0 dirs failed I0507 06:32:39.857108 11669 tablet_bootstrap.cc:439] T 826e8360d3e74edf8ea66f29380738ce P fa8f2d28bec94836a74c64cfb9ce6a54: Bootstrap starting. I0507 06:32:39.859591 11669 tablet_bootstrap.cc:590] T 826e8360d3e74edf8ea66f29380738ce P fa8f2d28bec94836a74c64cfb9ce6a54: No blocks or log segments found. Creating new log. I0507 06:32:39.864652 11669 log.cc:525] T 826e8360d3e74edf8ea66f29380738ce P fa8f2d28bec94836a74c64cfb9ce6a54: Log is configured to *not* fsync() on all Append() calls I0507 06:32:39.869860 11669 tablet_bootstrap.cc:439] T 826e8360d3e74edf8ea66f29380738ce P fa8f2d28bec94836a74c64cfb9ce6a54: No bootstrap required, opened a new log I0507 06:32:39.870170 11669 ts_tablet_manager.cc:1053] T 826e8360d3e74edf8ea66f29380738ce P fa8f2d28bec94836a74c64cfb9ce6a54: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.000s I0507 06:32:39.875998 11669 raft_consensus.cc:340] T 826e8360d3e74edf8ea66f29380738ce P fa8f2d28bec94836a74c64cfb9ce6a54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa8f2d28bec94836a74c64cfb9ce6a54" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 36629 } } I0507 06:32:39.876359 11669 raft_consensus.cc:366] T 826e8360d3e74edf8ea66f29380738ce P fa8f2d28bec94836a74c64cfb9ce6a54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0507 06:32:39.876421 11669 raft_consensus.cc:705] T 826e8360d3e74edf8ea66f29380738ce P fa8f2d28bec94836a74c64cfb9ce6a54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa8f2d28bec94836a74c64cfb9ce6a54, State: Initialized, Role: FOLLOWER I0507 06:32:39.876855 11669 consensus_queue.cc:229] T 826e8360d3e74edf8ea66f29380738ce P fa8f2d28bec94836a74c64cfb9ce6a54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa8f2d28bec94836a74c64cfb9ce6a54" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 36629 } } I0507 06:32:39.877025 11669 raft_consensus.cc:380] T 826e8360d3e74edf8ea66f29380738ce P fa8f2d28bec94836a74c64cfb9ce6a54 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0507 06:32:39.877091 11669 raft_consensus.cc:472] T 826e8360d3e74edf8ea66f29380738ce P fa8f2d28bec94836a74c64cfb9ce6a54 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0507 06:32:39.877169 11669 raft_consensus.cc:2886] T 826e8360d3e74edf8ea66f29380738ce P fa8f2d28bec94836a74c64cfb9ce6a54 [term 0 FOLLOWER]: Advancing to term 1 I0507 06:32:39.878859 11669 raft_consensus.cc:494] T 826e8360d3e74edf8ea66f29380738ce P fa8f2d28bec94836a74c64cfb9ce6a54 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa8f2d28bec94836a74c64cfb9ce6a54" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 36629 } } I0507 06:32:39.879237 11669 leader_election.cc:310] T 826e8360d3e74edf8ea66f29380738ce P fa8f2d28bec94836a74c64cfb9ce6a54 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fa8f2d28bec94836a74c64cfb9ce6a54; no voters: I0507 06:32:39.879956 11669 leader_election.cc:296] T 826e8360d3e74edf8ea66f29380738ce P fa8f2d28bec94836a74c64cfb9ce6a54 [CANDIDATE]: Term 1 election: Requested vote from peers I0507 06:32:39.880332 11671 raft_consensus.cc:2640] T 826e8360d3e74edf8ea66f29380738ce P fa8f2d28bec94836a74c64cfb9ce6a54 [term 1 FOLLOWER]: Leader election won for term 1 I0507 06:32:39.881052 11656 heartbeater.cc:467] Master 127.0.57.254:42725 was elected leader, sending a full tablet report... I0507 06:32:39.881798 11669 ts_tablet_manager.cc:1081] T 826e8360d3e74edf8ea66f29380738ce P fa8f2d28bec94836a74c64cfb9ce6a54: Time spent starting tablet: real 0.012s user 0.011s sys 0.001s I0507 06:32:39.882161 11671 raft_consensus.cc:667] T 826e8360d3e74edf8ea66f29380738ce P fa8f2d28bec94836a74c64cfb9ce6a54 [term 1 LEADER]: Becoming Leader. State: Replica: fa8f2d28bec94836a74c64cfb9ce6a54, State: Running, Role: LEADER I0507 06:32:39.882577 11671 consensus_queue.cc:206] T 826e8360d3e74edf8ea66f29380738ce P fa8f2d28bec94836a74c64cfb9ce6a54 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa8f2d28bec94836a74c64cfb9ce6a54" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 36629 } } I0507 06:32:39.887341 11491 catalog_manager.cc:3988] T 826e8360d3e74edf8ea66f29380738ce P fa8f2d28bec94836a74c64cfb9ce6a54 reported cstate change: term changed from 0 to 1, leader changed from to fa8f2d28bec94836a74c64cfb9ce6a54 (127.0.57.193). New cstate: current_term: 1 leader_uuid: "fa8f2d28bec94836a74c64cfb9ce6a54" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa8f2d28bec94836a74c64cfb9ce6a54" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 36629 } } } I0507 06:32:39.896575 231 external_mini_cluster.cc:623] 1 TS(s) registered with all masters I0507 06:32:39.897747 231 ts_itest-base.cc:232] Waiting for 1 tablets on tserver fa8f2d28bec94836a74c64cfb9ce6a54 to finish bootstrapping I0507 06:32:39.904929 231 external_mini_cluster.cc:1172] Attempting to check leaks for /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver pid 11526 I0507 06:32:39.905566 11653 generic_service.cc:171] Checking for leaks (request via RPC) I0507 06:32:39.956171 231 external_mini_cluster.cc:1131] Killing /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver with pid 11526 I0507 06:32:39.968577 231 external_mini_cluster.cc:1172] Attempting to check leaks for /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-master pid 11466 I0507 06:32:39.969786 11522 generic_service.cc:171] Checking for leaks (request via RPC) I0507 06:32:40.045315 231 external_mini_cluster.cc:1131] Killing /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-master with pid 11466 I0507 06:32:40.061614 231 external_mini_cluster.cc:933] Running /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-master /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-master --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_2.1557210500371863-231/raft_consensus-itest-cluster/master-0/wal --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_2.1557210500371863-231/raft_consensus-itest-cluster/master-0/data --block_manager=log --webserver_interface=localhost --ipki_ca_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.0.57.254:42725 --webserver_interface=127.0.57.254 --webserver_port=35977 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_2.1557210500371863-231/raft_consensus-itest-cluster/master-0/logs --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_2.1557210500371863-231/raft_consensus-itest-cluster/master-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=false --rpc_reuseport=true with env {} WARNING: Logging before InitGoogleLogging() is written to STDERR W0507 06:32:40.265251 11678 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0507 06:32:40.265488 11678 flags.cc:404] Enabled unsafe flag: --never_fsync=true W0507 06:32:40.267868 11678 flags.cc:404] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false W0507 06:32:40.267930 11678 flags.cc:404] Enabled experimental flag: --ipki_ca_key_size=1024 W0507 06:32:40.267961 11678 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W0507 06:32:40.268005 11678 flags.cc:404] Enabled experimental flag: --tsk_num_rsa_bits=512 W0507 06:32:40.268038 11678 flags.cc:404] Enabled experimental flag: --rpc_reuseport=true I0507 06:32:40.272704 11678 master_main.cc:73] Master server non-default flags: --raft_prepare_replacement_before_eviction=false --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_2.1557210500371863-231/raft_consensus-itest-cluster/master-0/data --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_2.1557210500371863-231/raft_consensus-itest-cluster/master-0/wal --ipki_ca_key_size=1024 --ipki_server_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.0.57.254:42725 --rpc_reuseport=true --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_2.1557210500371863-231/raft_consensus-itest-cluster/master-0/data/info.pb --webserver_interface=127.0.57.254 --webserver_port=35977 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_2.1557210500371863-231/raft_consensus-itest-cluster/master-0/logs --logbuflevel=-1 --logtostderr=true Master server version: kudu 1.10.0-cdh6.x-SNAPSHOT revision 7d0ee26697926e87af9ef395008799577f6ab43f build type FASTDEBUG built by jenkins at 06 May 2019 22:44:21 PST on quasar-knmpfx-1.vpc.cloudera.com build id 322017 ASAN enabled I0507 06:32:40.279022 11678 system_ntp.cc:202] NTP initialized. Skew: 500ppm Current error: 154516us I0507 06:32:40.280231 11678 fs_manager.cc:263] Metadata directory not provided I0507 06:32:40.280320 11678 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0507 06:32:40.286404 11678 fs_manager.cc:399] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s I0507 06:32:40.287047 11678 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0507 06:32:40.287605 11678 file_cache.cc:466] Constructed file cache lbm with capacity 419430 I0507 06:32:40.290774 11678 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s I0507 06:32:40.290876 11678 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_2.1557210500371863-231/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_2.1557210500371863-231/raft_consensus-itest-cluster/master-0/wal uuid: "4c6c527837014acf9fc19b31112f1a4a" format_stamp: "Formatted at 2019-05-07 06:32:39 on dist-test-slave-dist-test-slave-rqp6" I0507 06:32:40.291882 11678 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_2.1557210500371863-231/raft_consensus-itest-cluster/master-0/wal metadata directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_2.1557210500371863-231/raft_consensus-itest-cluster/master-0/wal 1 data directories: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_2.1557210500371863-231/raft_consensus-itest-cluster/master-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0507 06:32:40.334369 11678 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0507 06:32:40.350311 11678 rpc_server.cc:205] RPC server started. Bound to: 127.0.57.254:42725 I0507 06:32:40.351585 11678 webserver.cc:308] Webserver started at http://127.0.57.254:35977/ using document root and password file I0507 06:32:40.352542 11678 server_base.cc:608] Dumped server information to /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_2.1557210500371863-231/raft_consensus-itest-cluster/master-0/data/info.pb I0507 06:32:40.354921 231 external_mini_cluster.cc:995] Started /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-master as pid 11678 I0507 06:32:40.355463 231 external_mini_cluster.cc:933] Running /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_2.1557210500371863-231/raft_consensus-itest-cluster/ts-0/wal --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_2.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data --block_manager=log --rpc_bind_addresses=127.0.57.193:36629 --local_ip_for_outbound_sockets=127.0.57.193 --webserver_port=43791 --webserver_interface=127.0.57.193 --tserver_master_addrs=127.0.57.254:42725 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_2.1557210500371863-231/raft_consensus-itest-cluster/ts-0/logs --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_2.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --heartbeat_incompatible_replica_management_is_fatal=true --heartbeat_interval_ms=500 --raft_prepare_replacement_before_eviction=false --raft_prepare_replacement_before_eviction=true with env {} I0507 06:32:40.361146 11737 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 4c6c527837014acf9fc19b31112f1a4a: Bootstrap starting. I0507 06:32:40.390302 11737 log.cc:525] T 00000000000000000000000000000000 P 4c6c527837014acf9fc19b31112f1a4a: Log is configured to *not* fsync() on all Append() calls I0507 06:32:40.405930 11737 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 4c6c527837014acf9fc19b31112f1a4a: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=4 ignored=0} mutations{seen=2 ignored=0} orphaned_commits=0. Pending: 0 replicates I0507 06:32:40.406414 11737 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 4c6c527837014acf9fc19b31112f1a4a: Bootstrap complete. I0507 06:32:40.416066 11737 raft_consensus.cc:340] T 00000000000000000000000000000000 P 4c6c527837014acf9fc19b31112f1a4a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c6c527837014acf9fc19b31112f1a4a" member_type: VOTER } I0507 06:32:40.416863 11737 raft_consensus.cc:705] T 00000000000000000000000000000000 P 4c6c527837014acf9fc19b31112f1a4a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c6c527837014acf9fc19b31112f1a4a, State: Initialized, Role: FOLLOWER I0507 06:32:40.417404 11737 consensus_queue.cc:229] T 00000000000000000000000000000000 P 4c6c527837014acf9fc19b31112f1a4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c6c527837014acf9fc19b31112f1a4a" member_type: VOTER } I0507 06:32:40.417557 11737 raft_consensus.cc:380] T 00000000000000000000000000000000 P 4c6c527837014acf9fc19b31112f1a4a [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0507 06:32:40.417624 11737 raft_consensus.cc:472] T 00000000000000000000000000000000 P 4c6c527837014acf9fc19b31112f1a4a [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0507 06:32:40.417709 11737 raft_consensus.cc:2886] T 00000000000000000000000000000000 P 4c6c527837014acf9fc19b31112f1a4a [term 1 FOLLOWER]: Advancing to term 2 I0507 06:32:40.421651 11737 raft_consensus.cc:494] T 00000000000000000000000000000000 P 4c6c527837014acf9fc19b31112f1a4a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c6c527837014acf9fc19b31112f1a4a" member_type: VOTER } I0507 06:32:40.422063 11737 leader_election.cc:310] T 00000000000000000000000000000000 P 4c6c527837014acf9fc19b31112f1a4a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4c6c527837014acf9fc19b31112f1a4a; no voters: I0507 06:32:40.422842 11740 raft_consensus.cc:2640] T 00000000000000000000000000000000 P 4c6c527837014acf9fc19b31112f1a4a [term 2 FOLLOWER]: Leader election won for term 2 I0507 06:32:40.423161 11737 leader_election.cc:296] T 00000000000000000000000000000000 P 4c6c527837014acf9fc19b31112f1a4a [CANDIDATE]: Term 2 election: Requested vote from peers I0507 06:32:40.423961 11740 raft_consensus.cc:667] T 00000000000000000000000000000000 P 4c6c527837014acf9fc19b31112f1a4a [term 2 LEADER]: Becoming Leader. State: Replica: 4c6c527837014acf9fc19b31112f1a4a, State: Running, Role: LEADER I0507 06:32:40.424324 11740 consensus_queue.cc:206] T 00000000000000000000000000000000 P 4c6c527837014acf9fc19b31112f1a4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c6c527837014acf9fc19b31112f1a4a" member_type: VOTER } I0507 06:32:40.427471 11742 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4c6c527837014acf9fc19b31112f1a4a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4c6c527837014acf9fc19b31112f1a4a. Latest consensus state: current_term: 2 leader_uuid: "4c6c527837014acf9fc19b31112f1a4a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c6c527837014acf9fc19b31112f1a4a" member_type: VOTER } } I0507 06:32:40.427953 11742 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4c6c527837014acf9fc19b31112f1a4a [sys.catalog]: This master's current role is: LEADER I0507 06:32:40.428836 11737 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4c6c527837014acf9fc19b31112f1a4a [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 2 leader_uuid: "4c6c527837014acf9fc19b31112f1a4a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c6c527837014acf9fc19b31112f1a4a" member_type: VOTER } } I0507 06:32:40.428997 11737 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4c6c527837014acf9fc19b31112f1a4a [sys.catalog]: This master's current role is: LEADER I0507 06:32:40.428884 11741 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4c6c527837014acf9fc19b31112f1a4a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "4c6c527837014acf9fc19b31112f1a4a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c6c527837014acf9fc19b31112f1a4a" member_type: VOTER } } I0507 06:32:40.429158 11741 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4c6c527837014acf9fc19b31112f1a4a [sys.catalog]: This master's current role is: LEADER I0507 06:32:40.430058 11737 sys_catalog.cc:424] T 00000000000000000000000000000000 P 4c6c527837014acf9fc19b31112f1a4a [sys.catalog]: configured and running, proceeding with master startup. I0507 06:32:40.430708 11746 catalog_manager.cc:1055] Loading table and tablet metadata into memory... I0507 06:32:40.435346 11746 catalog_manager.cc:356] Loaded metadata for table TestTable [id=ad86553e031344aebd031dbb7c8b7bae] I0507 06:32:40.437149 11746 catalog_manager.cc:410] Loaded metadata for tablet 826e8360d3e74edf8ea66f29380738ce (table TestTable [id=ad86553e031344aebd031dbb7c8b7bae]) I0507 06:32:40.437346 11746 catalog_manager.cc:1066] Initializing Kudu internal certificate authority... I0507 06:32:40.440392 11746 catalog_manager.cc:1075] Loading token signing keys... I0507 06:32:40.441480 11746 catalog_manager.cc:4303] T 00000000000000000000000000000000 P 4c6c527837014acf9fc19b31112f1a4a: Loaded TSK: 0 WARNING: Logging before InitGoogleLogging() is written to STDERR W0507 06:32:40.541025 11738 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0507 06:32:40.541246 11738 flags.cc:404] Enabled unsafe flag: --never_fsync=true W0507 06:32:40.543349 11738 flags.cc:404] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true W0507 06:32:40.543409 11738 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W0507 06:32:40.543514 11738 flags.cc:404] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.57.193 I0507 06:32:40.549320 11738 tablet_server_main.cc:78] Tablet server non-default flags: --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_2.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_2.1557210500371863-231/raft_consensus-itest-cluster/ts-0/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.0.57.193:36629 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_2.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data/info.pb --webserver_interface=127.0.57.193 --webserver_port=43791 --heartbeat_interval_ms=500 --tserver_master_addrs=127.0.57.254:42725 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.0.57.193 --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_2.1557210500371863-231/raft_consensus-itest-cluster/ts-0/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.10.0-cdh6.x-SNAPSHOT revision 7d0ee26697926e87af9ef395008799577f6ab43f build type FASTDEBUG built by jenkins at 06 May 2019 22:44:21 PST on quasar-knmpfx-1.vpc.cloudera.com build id 322017 ASAN enabled I0507 06:32:40.557606 11738 system_ntp.cc:202] NTP initialized. Skew: 500ppm Current error: 154516us I0507 06:32:40.558331 11738 fs_manager.cc:263] Metadata directory not provided I0507 06:32:40.558409 11738 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0507 06:32:40.567286 11738 fs_manager.cc:399] Time spent opening directory manager: real 0.007s user 0.001s sys 0.008s I0507 06:32:40.568027 11738 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0507 06:32:40.568696 11738 file_cache.cc:466] Constructed file cache lbm with capacity 419430 I0507 06:32:40.571878 11738 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s I0507 06:32:40.571977 11738 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_2.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_2.1557210500371863-231/raft_consensus-itest-cluster/ts-0/wal uuid: "fa8f2d28bec94836a74c64cfb9ce6a54" format_stamp: "Formatted at 2019-05-07 06:32:39 on dist-test-slave-dist-test-slave-rqp6" I0507 06:32:40.573096 11738 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_2.1557210500371863-231/raft_consensus-itest-cluster/ts-0/wal metadata directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_2.1557210500371863-231/raft_consensus-itest-cluster/ts-0/wal 1 data directories: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_2.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0507 06:32:40.642891 11738 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0507 06:32:40.644862 11738 ts_tablet_manager.cc:344] Loading tablet metadata (0/1 complete) I0507 06:32:40.649336 11738 ts_tablet_manager.cc:356] Loaded tablet metadata (1 total tablets, 1 live tablets) I0507 06:32:40.649399 11738 ts_tablet_manager.cc:362] Registering tablets (0/1 complete) I0507 06:32:40.652752 11738 ts_tablet_manager.cc:376] Registered 1 tablets I0507 06:32:40.653524 11763 tablet_bootstrap.cc:439] T 826e8360d3e74edf8ea66f29380738ce P fa8f2d28bec94836a74c64cfb9ce6a54: Bootstrap starting. I0507 06:32:40.668401 11763 log.cc:525] T 826e8360d3e74edf8ea66f29380738ce P fa8f2d28bec94836a74c64cfb9ce6a54: Log is configured to *not* fsync() on all Append() calls I0507 06:32:40.671809 11763 tablet_bootstrap.cc:439] T 826e8360d3e74edf8ea66f29380738ce P fa8f2d28bec94836a74c64cfb9ce6a54: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I0507 06:32:40.672363 11763 tablet_bootstrap.cc:439] T 826e8360d3e74edf8ea66f29380738ce P fa8f2d28bec94836a74c64cfb9ce6a54: Bootstrap complete. I0507 06:32:40.673631 11763 ts_tablet_manager.cc:1053] T 826e8360d3e74edf8ea66f29380738ce P fa8f2d28bec94836a74c64cfb9ce6a54: Time spent bootstrapping tablet: real 0.020s user 0.011s sys 0.008s I0507 06:32:40.687376 11763 raft_consensus.cc:340] T 826e8360d3e74edf8ea66f29380738ce P fa8f2d28bec94836a74c64cfb9ce6a54 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa8f2d28bec94836a74c64cfb9ce6a54" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 36629 } } I0507 06:32:40.691431 11763 raft_consensus.cc:705] T 826e8360d3e74edf8ea66f29380738ce P fa8f2d28bec94836a74c64cfb9ce6a54 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa8f2d28bec94836a74c64cfb9ce6a54, State: Initialized, Role: FOLLOWER I0507 06:32:40.692065 11763 consensus_queue.cc:229] T 826e8360d3e74edf8ea66f29380738ce P fa8f2d28bec94836a74c64cfb9ce6a54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa8f2d28bec94836a74c64cfb9ce6a54" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 36629 } } I0507 06:32:40.692296 11763 raft_consensus.cc:380] T 826e8360d3e74edf8ea66f29380738ce P fa8f2d28bec94836a74c64cfb9ce6a54 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0507 06:32:40.692369 11763 raft_consensus.cc:472] T 826e8360d3e74edf8ea66f29380738ce P fa8f2d28bec94836a74c64cfb9ce6a54 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0507 06:32:40.692469 11763 raft_consensus.cc:2886] T 826e8360d3e74edf8ea66f29380738ce P fa8f2d28bec94836a74c64cfb9ce6a54 [term 1 FOLLOWER]: Advancing to term 2 I0507 06:32:40.695771 11763 raft_consensus.cc:494] T 826e8360d3e74edf8ea66f29380738ce P fa8f2d28bec94836a74c64cfb9ce6a54 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa8f2d28bec94836a74c64cfb9ce6a54" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 36629 } } I0507 06:32:40.696198 11763 leader_election.cc:310] T 826e8360d3e74edf8ea66f29380738ce P fa8f2d28bec94836a74c64cfb9ce6a54 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fa8f2d28bec94836a74c64cfb9ce6a54; no voters: I0507 06:32:40.697211 11763 leader_election.cc:296] T 826e8360d3e74edf8ea66f29380738ce P fa8f2d28bec94836a74c64cfb9ce6a54 [CANDIDATE]: Term 2 election: Requested vote from peers I0507 06:32:40.697469 11795 raft_consensus.cc:2640] T 826e8360d3e74edf8ea66f29380738ce P fa8f2d28bec94836a74c64cfb9ce6a54 [term 2 FOLLOWER]: Leader election won for term 2 I0507 06:32:40.698997 11763 ts_tablet_manager.cc:1081] T 826e8360d3e74edf8ea66f29380738ce P fa8f2d28bec94836a74c64cfb9ce6a54: Time spent starting tablet: real 0.025s user 0.021s sys 0.004s I0507 06:32:40.699611 11795 raft_consensus.cc:667] T 826e8360d3e74edf8ea66f29380738ce P fa8f2d28bec94836a74c64cfb9ce6a54 [term 2 LEADER]: Becoming Leader. State: Replica: fa8f2d28bec94836a74c64cfb9ce6a54, State: Running, Role: LEADER I0507 06:32:40.700042 11795 consensus_queue.cc:206] T 826e8360d3e74edf8ea66f29380738ce P fa8f2d28bec94836a74c64cfb9ce6a54 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa8f2d28bec94836a74c64cfb9ce6a54" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 36629 } } I0507 06:32:40.729959 11738 rpc_server.cc:205] RPC server started. Bound to: 127.0.57.193:36629 I0507 06:32:40.731279 11738 webserver.cc:308] Webserver started at http://127.0.57.193:43791/ using document root and password file I0507 06:32:40.731995 11738 server_base.cc:608] Dumped server information to /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_2.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data/info.pb I0507 06:32:40.733834 231 external_mini_cluster.cc:995] Started /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver as pid 11738 I0507 06:32:40.745393 11874 heartbeater.cc:345] Connected to a master server at 127.0.57.254:42725 I0507 06:32:40.745631 11874 heartbeater.cc:426] Registering TS with master... I0507 06:32:40.746161 11874 heartbeater.cc:475] Master 127.0.57.254:42725 requested a full tablet report, sending... W0507 06:32:40.748342 11704 master_service.cc:205] Configuration error: replica replacement scheme (PREPARE_REPLACEMENT_BEFORE_EVICTION) of the tablet server fa8f2d28bec94836a74c64cfb9ce6a54 at 127.0.57.193:56823 differs from the catalog manager's (EVICT_FIRST); they must be run with the same scheme (controlled by the --raft_prepare_replacement_before_eviction flag) W0507 06:32:40.749369 11874 heartbeater.cc:587] Failed to heartbeat to 127.0.57.254:42725 (0 consecutive failures): Configuration error: replica replacement scheme (PREPARE_REPLACEMENT_BEFORE_EVICTION) of the tablet server fa8f2d28bec94836a74c64cfb9ce6a54 at 127.0.57.193:56823 differs from the catalog manager's (EVICT_FIRST); they must be run with the same scheme (controlled by the --raft_prepare_replacement_before_eviction flag) F0507 06:32:40.749430 11874 heartbeater.cc:620] master detected incompatibility: Configuration error: replica replacement scheme (PREPARE_REPLACEMENT_BEFORE_EVICTION) of the tablet server fa8f2d28bec94836a74c64cfb9ce6a54 at 127.0.57.193:56823 differs from the catalog manager's (EVICT_FIRST); they must be run with the same scheme (controlled by the --raft_prepare_replacement_before_eviction flag) *** Check failure stack trace: *** @ 0x7efcb282131d google::LogMessage::Fail() at ??:0 @ 0x7efcb28231dd google::LogMessage::SendToLog() at ??:0 @ 0x7efcb2820e59 google::LogMessage::Flush() at ??:0 @ 0x7efcb2823c7f google::LogMessageFatal::~LogMessageFatal() at ??:0 @ 0x7efcc073246b kudu::tserver::Heartbeater::Thread::RunThread() at ??:0 @ 0x7efcc0745b83 boost::_bi::bind_t<>::operator()() at ??:0 @ 0x7efcb69cdd2c boost::function0<>::operator()() at ??:0 @ 0x7efcb3a7cc6b kudu::Thread::SuperviseThread() at ??:0 @ 0x7efcb9e1c184 start_thread at ??:0 @ 0x7efcaee50ffd clone at ??:0 I0507 06:32:42.452348 231 external_mini_cluster.cc:933] Running /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_2.1557210500371863-231/raft_consensus-itest-cluster/ts-0/wal --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_2.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data --block_manager=log --rpc_bind_addresses=127.0.57.193:36629 --local_ip_for_outbound_sockets=127.0.57.193 --webserver_port=43791 --webserver_interface=127.0.57.193 --tserver_master_addrs=127.0.57.254:42725 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_2.1557210500371863-231/raft_consensus-itest-cluster/ts-0/logs --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_2.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --heartbeat_incompatible_replica_management_is_fatal=true --heartbeat_interval_ms=500 --raft_prepare_replacement_before_eviction=false --heartbeat_inject_required_feature_flag=999 with env {} WARNING: Logging before InitGoogleLogging() is written to STDERR W0507 06:32:42.622251 11890 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0507 06:32:42.622490 11890 flags.cc:404] Enabled unsafe flag: --heartbeat_inject_required_feature_flag=999 W0507 06:32:42.622637 11890 flags.cc:404] Enabled unsafe flag: --never_fsync=true W0507 06:32:42.624778 11890 flags.cc:404] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false W0507 06:32:42.624857 11890 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W0507 06:32:42.624965 11890 flags.cc:404] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.57.193 I0507 06:32:42.629148 11890 tablet_server_main.cc:78] Tablet server non-default flags: --raft_prepare_replacement_before_eviction=false --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_2.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_2.1557210500371863-231/raft_consensus-itest-cluster/ts-0/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.0.57.193:36629 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_2.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data/info.pb --webserver_interface=127.0.57.193 --webserver_port=43791 --heartbeat_inject_required_feature_flag=999 --heartbeat_interval_ms=500 --tserver_master_addrs=127.0.57.254:42725 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.0.57.193 --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_2.1557210500371863-231/raft_consensus-itest-cluster/ts-0/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.10.0-cdh6.x-SNAPSHOT revision 7d0ee26697926e87af9ef395008799577f6ab43f build type FASTDEBUG built by jenkins at 06 May 2019 22:44:21 PST on quasar-knmpfx-1.vpc.cloudera.com build id 322017 ASAN enabled I0507 06:32:42.637121 11890 system_ntp.cc:202] NTP initialized. Skew: 500ppm Current error: 155516us I0507 06:32:42.637820 11890 fs_manager.cc:263] Metadata directory not provided I0507 06:32:42.637899 11890 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0507 06:32:42.643869 11890 fs_manager.cc:399] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s I0507 06:32:42.644589 11890 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0507 06:32:42.645164 11890 file_cache.cc:466] Constructed file cache lbm with capacity 419430 I0507 06:32:42.648705 11890 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s I0507 06:32:42.648787 11890 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_2.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_2.1557210500371863-231/raft_consensus-itest-cluster/ts-0/wal uuid: "fa8f2d28bec94836a74c64cfb9ce6a54" format_stamp: "Formatted at 2019-05-07 06:32:39 on dist-test-slave-dist-test-slave-rqp6" I0507 06:32:42.649794 11890 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_2.1557210500371863-231/raft_consensus-itest-cluster/ts-0/wal metadata directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_2.1557210500371863-231/raft_consensus-itest-cluster/ts-0/wal 1 data directories: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_2.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0507 06:32:42.674697 11890 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0507 06:32:42.676666 11890 ts_tablet_manager.cc:344] Loading tablet metadata (0/1 complete) I0507 06:32:42.680384 11890 ts_tablet_manager.cc:356] Loaded tablet metadata (1 total tablets, 1 live tablets) I0507 06:32:42.680446 11890 ts_tablet_manager.cc:362] Registering tablets (0/1 complete) I0507 06:32:42.683096 11890 ts_tablet_manager.cc:376] Registered 1 tablets I0507 06:32:42.683691 11905 tablet_bootstrap.cc:439] T 826e8360d3e74edf8ea66f29380738ce P fa8f2d28bec94836a74c64cfb9ce6a54: Bootstrap starting. I0507 06:32:42.700744 11905 log.cc:525] T 826e8360d3e74edf8ea66f29380738ce P fa8f2d28bec94836a74c64cfb9ce6a54: Log is configured to *not* fsync() on all Append() calls I0507 06:32:42.706374 11905 tablet_bootstrap.cc:439] T 826e8360d3e74edf8ea66f29380738ce P fa8f2d28bec94836a74c64cfb9ce6a54: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I0507 06:32:42.707010 11905 tablet_bootstrap.cc:439] T 826e8360d3e74edf8ea66f29380738ce P fa8f2d28bec94836a74c64cfb9ce6a54: Bootstrap complete. I0507 06:32:42.708225 11905 ts_tablet_manager.cc:1053] T 826e8360d3e74edf8ea66f29380738ce P fa8f2d28bec94836a74c64cfb9ce6a54: Time spent bootstrapping tablet: real 0.025s user 0.010s sys 0.011s I0507 06:32:42.722625 11905 raft_consensus.cc:340] T 826e8360d3e74edf8ea66f29380738ce P fa8f2d28bec94836a74c64cfb9ce6a54 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa8f2d28bec94836a74c64cfb9ce6a54" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 36629 } } I0507 06:32:42.725378 11905 raft_consensus.cc:705] T 826e8360d3e74edf8ea66f29380738ce P fa8f2d28bec94836a74c64cfb9ce6a54 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa8f2d28bec94836a74c64cfb9ce6a54, State: Initialized, Role: FOLLOWER I0507 06:32:42.725917 11905 consensus_queue.cc:229] T 826e8360d3e74edf8ea66f29380738ce P fa8f2d28bec94836a74c64cfb9ce6a54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa8f2d28bec94836a74c64cfb9ce6a54" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 36629 } } I0507 06:32:42.726130 11905 raft_consensus.cc:380] T 826e8360d3e74edf8ea66f29380738ce P fa8f2d28bec94836a74c64cfb9ce6a54 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0507 06:32:42.726197 11905 raft_consensus.cc:472] T 826e8360d3e74edf8ea66f29380738ce P fa8f2d28bec94836a74c64cfb9ce6a54 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0507 06:32:42.726297 11905 raft_consensus.cc:2886] T 826e8360d3e74edf8ea66f29380738ce P fa8f2d28bec94836a74c64cfb9ce6a54 [term 2 FOLLOWER]: Advancing to term 3 I0507 06:32:42.729482 11905 raft_consensus.cc:494] T 826e8360d3e74edf8ea66f29380738ce P fa8f2d28bec94836a74c64cfb9ce6a54 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa8f2d28bec94836a74c64cfb9ce6a54" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 36629 } } I0507 06:32:42.729897 11905 leader_election.cc:310] T 826e8360d3e74edf8ea66f29380738ce P fa8f2d28bec94836a74c64cfb9ce6a54 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fa8f2d28bec94836a74c64cfb9ce6a54; no voters: I0507 06:32:42.731026 11905 leader_election.cc:296] T 826e8360d3e74edf8ea66f29380738ce P fa8f2d28bec94836a74c64cfb9ce6a54 [CANDIDATE]: Term 3 election: Requested vote from peers I0507 06:32:42.731287 11974 raft_consensus.cc:2640] T 826e8360d3e74edf8ea66f29380738ce P fa8f2d28bec94836a74c64cfb9ce6a54 [term 3 FOLLOWER]: Leader election won for term 3 I0507 06:32:42.732777 11905 ts_tablet_manager.cc:1081] T 826e8360d3e74edf8ea66f29380738ce P fa8f2d28bec94836a74c64cfb9ce6a54: Time spent starting tablet: real 0.024s user 0.018s sys 0.003s I0507 06:32:42.732661 11974 raft_consensus.cc:667] T 826e8360d3e74edf8ea66f29380738ce P fa8f2d28bec94836a74c64cfb9ce6a54 [term 3 LEADER]: Becoming Leader. State: Replica: fa8f2d28bec94836a74c64cfb9ce6a54, State: Running, Role: LEADER I0507 06:32:42.733655 11974 consensus_queue.cc:206] T 826e8360d3e74edf8ea66f29380738ce P fa8f2d28bec94836a74c64cfb9ce6a54 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa8f2d28bec94836a74c64cfb9ce6a54" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 36629 } } I0507 06:32:42.743600 11890 rpc_server.cc:205] RPC server started. Bound to: 127.0.57.193:36629 I0507 06:32:42.744837 11890 webserver.cc:308] Webserver started at http://127.0.57.193:43791/ using document root and password file I0507 06:32:42.745537 11890 server_base.cc:608] Dumped server information to /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_2.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data/info.pb I0507 06:32:42.751804 231 external_mini_cluster.cc:995] Started /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver as pid 11890 I0507 06:32:42.755050 12016 heartbeater.cc:345] Connected to a master server at 127.0.57.254:42725 I0507 06:32:42.755271 12016 heartbeater.cc:426] Registering TS with master... I0507 06:32:42.755765 12016 heartbeater.cc:475] Master 127.0.57.254:42725 requested a full tablet report, sending... W0507 06:32:42.757055 11685 messenger.cc:389] Unable to handle RPC call: Not implemented: call requires unsupported application feature flags: 999 W0507 06:32:42.757577 12016 heartbeater.cc:587] Failed to heartbeat to 127.0.57.254:42725 (0 consecutive failures): Remote error: Failed to send heartbeat to master: unsupported feature flags F0507 06:32:42.757665 12016 heartbeater.cc:620] master does not support required feature flags: Remote error: Failed to send heartbeat to master: unsupported feature flags *** Check failure stack trace: *** @ 0x7f754ea9531d google::LogMessage::Fail() at ??:0 @ 0x7f754ea971dd google::LogMessage::SendToLog() at ??:0 @ 0x7f754ea94e59 google::LogMessage::Flush() at ??:0 @ 0x7f754ea97c7f google::LogMessageFatal::~LogMessageFatal() at ??:0 @ 0x7f755c9a646b kudu::tserver::Heartbeater::Thread::RunThread() at ??:0 @ 0x7f755c9b9b83 boost::_bi::bind_t<>::operator()() at ??:0 @ 0x7f7552c41d2c boost::function0<>::operator()() at ??:0 @ 0x7f754fcf0c6b kudu::Thread::SuperviseThread() at ??:0 @ 0x7f7556090184 start_thread at ??:0 @ 0x7f754b0c4ffd clone at ??:0 I0507 06:32:44.465817 231 external_mini_cluster.cc:1172] Attempting to check leaks for /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-master pid 11678 I0507 06:32:44.466547 11734 generic_service.cc:171] Checking for leaks (request via RPC) I0507 06:32:44.546262 231 external_mini_cluster.cc:1131] Killing /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-master with pid 11678 [ OK ] IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig/2 (5384 ms) [ RUN ] IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig/3 I0507 06:32:44.568445 231 test_util.cc:212] Using random seed: 1651926875 I0507 06:32:44.570005 231 ts_itest-base.cc:112] Starting cluster with: I0507 06:32:44.570052 231 ts_itest-base.cc:113] -------------- I0507 06:32:44.570067 231 ts_itest-base.cc:114] 1 tablet servers I0507 06:32:44.570092 231 ts_itest-base.cc:115] 1 replicas per TS I0507 06:32:44.570130 231 ts_itest-base.cc:116] -------------- I0507 06:32:44.571234 231 external_mini_cluster.cc:933] Running /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-master /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-master --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_3.1557210500371863-231/raft_consensus-itest-cluster/master-0/wal --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_3.1557210500371863-231/raft_consensus-itest-cluster/master-0/data --block_manager=log --webserver_interface=localhost --ipki_ca_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.0.57.254:37263 --webserver_interface=127.0.57.254 --webserver_port=0 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_3.1557210500371863-231/raft_consensus-itest-cluster/master-0/logs --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_3.1557210500371863-231/raft_consensus-itest-cluster/master-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=true --rpc_reuseport=true with env {} WARNING: Logging before InitGoogleLogging() is written to STDERR W0507 06:32:44.773666 12035 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0507 06:32:44.773876 12035 flags.cc:404] Enabled unsafe flag: --never_fsync=true W0507 06:32:44.776244 12035 flags.cc:404] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true W0507 06:32:44.776301 12035 flags.cc:404] Enabled experimental flag: --ipki_ca_key_size=1024 W0507 06:32:44.776332 12035 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W0507 06:32:44.776362 12035 flags.cc:404] Enabled experimental flag: --tsk_num_rsa_bits=512 W0507 06:32:44.776392 12035 flags.cc:404] Enabled experimental flag: --rpc_reuseport=true I0507 06:32:44.781313 12035 master_main.cc:73] Master server non-default flags: --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_3.1557210500371863-231/raft_consensus-itest-cluster/master-0/data --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_3.1557210500371863-231/raft_consensus-itest-cluster/master-0/wal --ipki_ca_key_size=1024 --ipki_server_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.0.57.254:37263 --rpc_reuseport=true --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_3.1557210500371863-231/raft_consensus-itest-cluster/master-0/data/info.pb --webserver_interface=127.0.57.254 --webserver_port=0 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_3.1557210500371863-231/raft_consensus-itest-cluster/master-0/logs --logbuflevel=-1 --logtostderr=true Master server version: kudu 1.10.0-cdh6.x-SNAPSHOT revision 7d0ee26697926e87af9ef395008799577f6ab43f build type FASTDEBUG built by jenkins at 06 May 2019 22:44:21 PST on quasar-knmpfx-1.vpc.cloudera.com build id 322017 ASAN enabled I0507 06:32:44.789119 12035 system_ntp.cc:202] NTP initialized. Skew: 500ppm Current error: 156516us I0507 06:32:44.790710 12035 fs_manager.cc:263] Metadata directory not provided I0507 06:32:44.790788 12035 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0507 06:32:44.791180 12035 server_base.cc:433] Could not load existing FS layout: Not found: could not find a healthy instance file I0507 06:32:44.791262 12035 server_base.cc:434] Attempting to create new FS layout instead I0507 06:32:44.794144 12035 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_3.1557210500371863-231/raft_consensus-itest-cluster/master-0/data/instance: uuid: "e3246a64546142049e1d917a1380a9e1" format_stamp: "Formatted at 2019-05-07 06:32:44 on dist-test-slave-dist-test-slave-rqp6" I0507 06:32:44.794749 12035 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_3.1557210500371863-231/raft_consensus-itest-cluster/master-0/wal/instance: uuid: "e3246a64546142049e1d917a1380a9e1" format_stamp: "Formatted at 2019-05-07 06:32:44 on dist-test-slave-dist-test-slave-rqp6" I0507 06:32:44.802412 12035 fs_manager.cc:503] Time spent creating directory manager: real 0.007s user 0.007s sys 0.001s I0507 06:32:44.803586 12035 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0507 06:32:44.804237 12035 file_cache.cc:466] Constructed file cache lbm with capacity 419430 I0507 06:32:44.807646 12035 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s I0507 06:32:44.807797 12035 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_3.1557210500371863-231/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_3.1557210500371863-231/raft_consensus-itest-cluster/master-0/wal uuid: "e3246a64546142049e1d917a1380a9e1" format_stamp: "Formatted at 2019-05-07 06:32:44 on dist-test-slave-dist-test-slave-rqp6" I0507 06:32:44.807983 12035 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_3.1557210500371863-231/raft_consensus-itest-cluster/master-0/wal metadata directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_3.1557210500371863-231/raft_consensus-itest-cluster/master-0/wal 1 data directories: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_3.1557210500371863-231/raft_consensus-itest-cluster/master-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0507 06:32:44.834300 12035 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0507 06:32:44.849642 12035 rpc_server.cc:205] RPC server started. Bound to: 127.0.57.254:37263 I0507 06:32:44.851074 12035 webserver.cc:308] Webserver started at http://127.0.57.254:42225/ using document root and password file I0507 06:32:44.852003 12035 server_base.cc:608] Dumped server information to /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_3.1557210500371863-231/raft_consensus-itest-cluster/master-0/data/info.pb I0507 06:32:44.853328 231 external_mini_cluster.cc:995] Started /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-master as pid 12035 I0507 06:32:44.854279 231 external_mini_cluster.cc:933] Running /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_3.1557210500371863-231/raft_consensus-itest-cluster/ts-0/wal --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_3.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data --block_manager=log --rpc_bind_addresses=127.0.57.193:0 --local_ip_for_outbound_sockets=127.0.57.193 --webserver_interface=127.0.57.193 --webserver_port=0 --tserver_master_addrs=127.0.57.254:37263 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_3.1557210500371863-231/raft_consensus-itest-cluster/ts-0/logs --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_3.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --heartbeat_incompatible_replica_management_is_fatal=true --heartbeat_interval_ms=500 --raft_prepare_replacement_before_eviction=true with env {} I0507 06:32:44.862370 12094 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0507 06:32:44.873560 12094 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P e3246a64546142049e1d917a1380a9e1: Bootstrap starting. I0507 06:32:44.876215 12094 tablet_bootstrap.cc:590] T 00000000000000000000000000000000 P e3246a64546142049e1d917a1380a9e1: No blocks or log segments found. Creating new log. I0507 06:32:44.877182 12094 log.cc:525] T 00000000000000000000000000000000 P e3246a64546142049e1d917a1380a9e1: Log is configured to *not* fsync() on all Append() calls I0507 06:32:44.882231 12094 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P e3246a64546142049e1d917a1380a9e1: No bootstrap required, opened a new log I0507 06:32:44.888743 12094 raft_consensus.cc:340] T 00000000000000000000000000000000 P e3246a64546142049e1d917a1380a9e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3246a64546142049e1d917a1380a9e1" member_type: VOTER } I0507 06:32:44.889027 12094 raft_consensus.cc:366] T 00000000000000000000000000000000 P e3246a64546142049e1d917a1380a9e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0507 06:32:44.889122 12094 raft_consensus.cc:705] T 00000000000000000000000000000000 P e3246a64546142049e1d917a1380a9e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3246a64546142049e1d917a1380a9e1, State: Initialized, Role: FOLLOWER I0507 06:32:44.889766 12094 consensus_queue.cc:229] T 00000000000000000000000000000000 P e3246a64546142049e1d917a1380a9e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3246a64546142049e1d917a1380a9e1" member_type: VOTER } I0507 06:32:44.889971 12094 raft_consensus.cc:380] T 00000000000000000000000000000000 P e3246a64546142049e1d917a1380a9e1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0507 06:32:44.890050 12094 raft_consensus.cc:472] T 00000000000000000000000000000000 P e3246a64546142049e1d917a1380a9e1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0507 06:32:44.890136 12094 raft_consensus.cc:2886] T 00000000000000000000000000000000 P e3246a64546142049e1d917a1380a9e1 [term 0 FOLLOWER]: Advancing to term 1 I0507 06:32:44.891721 12094 raft_consensus.cc:494] T 00000000000000000000000000000000 P e3246a64546142049e1d917a1380a9e1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3246a64546142049e1d917a1380a9e1" member_type: VOTER } I0507 06:32:44.892154 12094 leader_election.cc:310] T 00000000000000000000000000000000 P e3246a64546142049e1d917a1380a9e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e3246a64546142049e1d917a1380a9e1; no voters: I0507 06:32:44.893113 12097 raft_consensus.cc:2640] T 00000000000000000000000000000000 P e3246a64546142049e1d917a1380a9e1 [term 1 FOLLOWER]: Leader election won for term 1 I0507 06:32:44.893431 12094 leader_election.cc:296] T 00000000000000000000000000000000 P e3246a64546142049e1d917a1380a9e1 [CANDIDATE]: Term 1 election: Requested vote from peers I0507 06:32:44.894248 12097 raft_consensus.cc:667] T 00000000000000000000000000000000 P e3246a64546142049e1d917a1380a9e1 [term 1 LEADER]: Becoming Leader. State: Replica: e3246a64546142049e1d917a1380a9e1, State: Running, Role: LEADER I0507 06:32:44.894655 12097 consensus_queue.cc:206] T 00000000000000000000000000000000 P e3246a64546142049e1d917a1380a9e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3246a64546142049e1d917a1380a9e1" member_type: VOTER } I0507 06:32:44.897936 12099 sys_catalog.cc:337] T 00000000000000000000000000000000 P e3246a64546142049e1d917a1380a9e1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e3246a64546142049e1d917a1380a9e1. Latest consensus state: current_term: 1 leader_uuid: "e3246a64546142049e1d917a1380a9e1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3246a64546142049e1d917a1380a9e1" member_type: VOTER } } I0507 06:32:44.898475 12099 sys_catalog.cc:340] T 00000000000000000000000000000000 P e3246a64546142049e1d917a1380a9e1 [sys.catalog]: This master's current role is: LEADER I0507 06:32:44.898667 12094 sys_catalog.cc:337] T 00000000000000000000000000000000 P e3246a64546142049e1d917a1380a9e1 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 1 leader_uuid: "e3246a64546142049e1d917a1380a9e1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3246a64546142049e1d917a1380a9e1" member_type: VOTER } } I0507 06:32:44.899102 12094 sys_catalog.cc:340] T 00000000000000000000000000000000 P e3246a64546142049e1d917a1380a9e1 [sys.catalog]: This master's current role is: LEADER I0507 06:32:44.899020 12098 sys_catalog.cc:337] T 00000000000000000000000000000000 P e3246a64546142049e1d917a1380a9e1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e3246a64546142049e1d917a1380a9e1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3246a64546142049e1d917a1380a9e1" member_type: VOTER } } I0507 06:32:44.900203 12098 sys_catalog.cc:340] T 00000000000000000000000000000000 P e3246a64546142049e1d917a1380a9e1 [sys.catalog]: This master's current role is: LEADER I0507 06:32:44.901295 12094 sys_catalog.cc:424] T 00000000000000000000000000000000 P e3246a64546142049e1d917a1380a9e1 [sys.catalog]: configured and running, proceeding with master startup. I0507 06:32:44.901710 12103 catalog_manager.cc:1055] Loading table and tablet metadata into memory... I0507 06:32:44.901877 12104 catalog_manager.cc:538] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting I0507 06:32:44.906010 12103 catalog_manager.cc:1066] Initializing Kudu internal certificate authority... I0507 06:32:44.975144 12103 catalog_manager.cc:950] Generated new certificate authority record I0507 06:32:44.978960 12103 catalog_manager.cc:1075] Loading token signing keys... I0507 06:32:44.990547 12103 catalog_manager.cc:4292] T 00000000000000000000000000000000 P e3246a64546142049e1d917a1380a9e1: Generated new TSK 0 WARNING: Logging before InitGoogleLogging() is written to STDERR W0507 06:32:45.034495 12095 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0507 06:32:45.034740 12095 flags.cc:404] Enabled unsafe flag: --never_fsync=true W0507 06:32:45.036907 12095 flags.cc:404] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true W0507 06:32:45.036983 12095 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W0507 06:32:45.037092 12095 flags.cc:404] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.57.193 I0507 06:32:45.052981 12095 tablet_server_main.cc:78] Tablet server non-default flags: --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_3.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_3.1557210500371863-231/raft_consensus-itest-cluster/ts-0/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.0.57.193:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_3.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data/info.pb --webserver_interface=127.0.57.193 --webserver_port=0 --heartbeat_interval_ms=500 --tserver_master_addrs=127.0.57.254:37263 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.0.57.193 --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_3.1557210500371863-231/raft_consensus-itest-cluster/ts-0/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.10.0-cdh6.x-SNAPSHOT revision 7d0ee26697926e87af9ef395008799577f6ab43f build type FASTDEBUG built by jenkins at 06 May 2019 22:44:21 PST on quasar-knmpfx-1.vpc.cloudera.com build id 322017 ASAN enabled I0507 06:32:45.061374 12095 system_ntp.cc:202] NTP initialized. Skew: 500ppm Current error: 157016us I0507 06:32:45.062119 12095 fs_manager.cc:263] Metadata directory not provided I0507 06:32:45.062181 12095 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0507 06:32:45.062503 12095 server_base.cc:433] Could not load existing FS layout: Not found: could not find a healthy instance file I0507 06:32:45.062578 12095 server_base.cc:434] Attempting to create new FS layout instead I0507 06:32:45.065374 12095 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_3.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data/instance: uuid: "a82d40cf7c44462e88ae97100184ca36" format_stamp: "Formatted at 2019-05-07 06:32:45 on dist-test-slave-dist-test-slave-rqp6" I0507 06:32:45.066263 12095 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_3.1557210500371863-231/raft_consensus-itest-cluster/ts-0/wal/instance: uuid: "a82d40cf7c44462e88ae97100184ca36" format_stamp: "Formatted at 2019-05-07 06:32:45 on dist-test-slave-dist-test-slave-rqp6" I0507 06:32:45.075165 12095 fs_manager.cc:503] Time spent creating directory manager: real 0.009s user 0.007s sys 0.001s I0507 06:32:45.076375 12095 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0507 06:32:45.077123 12095 file_cache.cc:466] Constructed file cache lbm with capacity 419430 I0507 06:32:45.082458 12095 fs_manager.cc:419] Time spent opening block manager: real 0.004s user 0.000s sys 0.003s I0507 06:32:45.082556 12095 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_3.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_3.1557210500371863-231/raft_consensus-itest-cluster/ts-0/wal uuid: "a82d40cf7c44462e88ae97100184ca36" format_stamp: "Formatted at 2019-05-07 06:32:45 on dist-test-slave-dist-test-slave-rqp6" I0507 06:32:45.082726 12095 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_3.1557210500371863-231/raft_consensus-itest-cluster/ts-0/wal metadata directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_3.1557210500371863-231/raft_consensus-itest-cluster/ts-0/wal 1 data directories: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_3.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0507 06:32:45.125363 12095 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0507 06:32:45.137866 12095 ts_tablet_manager.cc:356] Loaded tablet metadata (0 total tablets, 0 live tablets) I0507 06:32:45.137961 12095 ts_tablet_manager.cc:376] Registered 0 tablets I0507 06:32:45.198207 12095 rpc_server.cc:205] RPC server started. Bound to: 127.0.57.193:39509 I0507 06:32:45.199523 12095 webserver.cc:308] Webserver started at http://127.0.57.193:46469/ using document root and password file I0507 06:32:45.200325 12095 server_base.cc:608] Dumped server information to /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_3.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data/info.pb I0507 06:32:45.202551 231 external_mini_cluster.cc:995] Started /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver as pid 12095 I0507 06:32:45.227006 12225 heartbeater.cc:345] Connected to a master server at 127.0.57.254:37263 I0507 06:32:45.227254 12225 heartbeater.cc:426] Registering TS with master... I0507 06:32:45.227833 12225 heartbeater.cc:475] Master 127.0.57.254:37263 requested a full tablet report, sending... I0507 06:32:45.229588 12060 ts_manager.cc:159] Registered new tserver with Master: a82d40cf7c44462e88ae97100184ca36 (127.0.57.193:39509) I0507 06:32:45.231215 12060 master_service.cc:273] Signed X509 certificate for tserver {username='slave'} at 127.0.57.193:40391 I0507 06:32:45.231727 231 external_mini_cluster.cc:623] 1 TS(s) registered with all masters I0507 06:32:45.243773 12060 catalog_manager.cc:1415] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46738: name: "TestTable" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 1 split_rows_range_bounds { } partition_schema { range_schema { columns { name: "key" } } } I0507 06:32:45.273088 12162 tablet_service.cc:1016] Processing CreateTablet for tablet 9ad35469733c4e7cbeafc3da5cdbbbfc (table=TestTable [id=4efe84c6df5a44f4b051d1c093c35f57]), partition=RANGE (key) PARTITION UNBOUNDED I0507 06:32:45.274391 12162 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet 9ad35469733c4e7cbeafc3da5cdbbbfc. 1 dirs total, 0 dirs full, 0 dirs failed I0507 06:32:45.286075 12238 tablet_bootstrap.cc:439] T 9ad35469733c4e7cbeafc3da5cdbbbfc P a82d40cf7c44462e88ae97100184ca36: Bootstrap starting. I0507 06:32:45.288859 12238 tablet_bootstrap.cc:590] T 9ad35469733c4e7cbeafc3da5cdbbbfc P a82d40cf7c44462e88ae97100184ca36: No blocks or log segments found. Creating new log. I0507 06:32:45.289878 12238 log.cc:525] T 9ad35469733c4e7cbeafc3da5cdbbbfc P a82d40cf7c44462e88ae97100184ca36: Log is configured to *not* fsync() on all Append() calls I0507 06:32:45.291597 12238 tablet_bootstrap.cc:439] T 9ad35469733c4e7cbeafc3da5cdbbbfc P a82d40cf7c44462e88ae97100184ca36: No bootstrap required, opened a new log I0507 06:32:45.291875 12238 ts_tablet_manager.cc:1053] T 9ad35469733c4e7cbeafc3da5cdbbbfc P a82d40cf7c44462e88ae97100184ca36: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.005s I0507 06:32:45.297619 12238 raft_consensus.cc:340] T 9ad35469733c4e7cbeafc3da5cdbbbfc P a82d40cf7c44462e88ae97100184ca36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a82d40cf7c44462e88ae97100184ca36" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 39509 } } I0507 06:32:45.297991 12238 raft_consensus.cc:366] T 9ad35469733c4e7cbeafc3da5cdbbbfc P a82d40cf7c44462e88ae97100184ca36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0507 06:32:45.298070 12238 raft_consensus.cc:705] T 9ad35469733c4e7cbeafc3da5cdbbbfc P a82d40cf7c44462e88ae97100184ca36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a82d40cf7c44462e88ae97100184ca36, State: Initialized, Role: FOLLOWER I0507 06:32:45.298538 12238 consensus_queue.cc:229] T 9ad35469733c4e7cbeafc3da5cdbbbfc P a82d40cf7c44462e88ae97100184ca36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a82d40cf7c44462e88ae97100184ca36" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 39509 } } I0507 06:32:45.298719 12238 raft_consensus.cc:380] T 9ad35469733c4e7cbeafc3da5cdbbbfc P a82d40cf7c44462e88ae97100184ca36 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0507 06:32:45.298791 12238 raft_consensus.cc:472] T 9ad35469733c4e7cbeafc3da5cdbbbfc P a82d40cf7c44462e88ae97100184ca36 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0507 06:32:45.298884 12238 raft_consensus.cc:2886] T 9ad35469733c4e7cbeafc3da5cdbbbfc P a82d40cf7c44462e88ae97100184ca36 [term 0 FOLLOWER]: Advancing to term 1 I0507 06:32:45.300603 12238 raft_consensus.cc:494] T 9ad35469733c4e7cbeafc3da5cdbbbfc P a82d40cf7c44462e88ae97100184ca36 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a82d40cf7c44462e88ae97100184ca36" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 39509 } } I0507 06:32:45.301030 12238 leader_election.cc:310] T 9ad35469733c4e7cbeafc3da5cdbbbfc P a82d40cf7c44462e88ae97100184ca36 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a82d40cf7c44462e88ae97100184ca36; no voters: I0507 06:32:45.301666 12238 leader_election.cc:296] T 9ad35469733c4e7cbeafc3da5cdbbbfc P a82d40cf7c44462e88ae97100184ca36 [CANDIDATE]: Term 1 election: Requested vote from peers I0507 06:32:45.302187 12240 raft_consensus.cc:2640] T 9ad35469733c4e7cbeafc3da5cdbbbfc P a82d40cf7c44462e88ae97100184ca36 [term 1 FOLLOWER]: Leader election won for term 1 I0507 06:32:45.302711 12225 heartbeater.cc:467] Master 127.0.57.254:37263 was elected leader, sending a full tablet report... I0507 06:32:45.303117 12238 ts_tablet_manager.cc:1081] T 9ad35469733c4e7cbeafc3da5cdbbbfc P a82d40cf7c44462e88ae97100184ca36: Time spent starting tablet: real 0.011s user 0.012s sys 0.000s I0507 06:32:45.303484 12240 raft_consensus.cc:667] T 9ad35469733c4e7cbeafc3da5cdbbbfc P a82d40cf7c44462e88ae97100184ca36 [term 1 LEADER]: Becoming Leader. State: Replica: a82d40cf7c44462e88ae97100184ca36, State: Running, Role: LEADER I0507 06:32:45.303875 12240 consensus_queue.cc:206] T 9ad35469733c4e7cbeafc3da5cdbbbfc P a82d40cf7c44462e88ae97100184ca36 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a82d40cf7c44462e88ae97100184ca36" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 39509 } } I0507 06:32:45.307907 12060 catalog_manager.cc:3988] T 9ad35469733c4e7cbeafc3da5cdbbbfc P a82d40cf7c44462e88ae97100184ca36 reported cstate change: term changed from 0 to 1, leader changed from to a82d40cf7c44462e88ae97100184ca36 (127.0.57.193). New cstate: current_term: 1 leader_uuid: "a82d40cf7c44462e88ae97100184ca36" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a82d40cf7c44462e88ae97100184ca36" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 39509 } health_report { overall_health: HEALTHY } } } I0507 06:32:45.325451 231 external_mini_cluster.cc:623] 1 TS(s) registered with all masters I0507 06:32:45.326856 231 ts_itest-base.cc:232] Waiting for 1 tablets on tserver a82d40cf7c44462e88ae97100184ca36 to finish bootstrapping I0507 06:32:45.334501 231 external_mini_cluster.cc:1172] Attempting to check leaks for /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver pid 12095 I0507 06:32:45.335192 12222 generic_service.cc:171] Checking for leaks (request via RPC) I0507 06:32:45.387512 231 external_mini_cluster.cc:1131] Killing /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver with pid 12095 I0507 06:32:45.398257 231 external_mini_cluster.cc:1172] Attempting to check leaks for /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-master pid 12035 I0507 06:32:45.399420 12091 generic_service.cc:171] Checking for leaks (request via RPC) I0507 06:32:45.484098 231 external_mini_cluster.cc:1131] Killing /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-master with pid 12035 I0507 06:32:45.498286 231 external_mini_cluster.cc:933] Running /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-master /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-master --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_3.1557210500371863-231/raft_consensus-itest-cluster/master-0/wal --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_3.1557210500371863-231/raft_consensus-itest-cluster/master-0/data --block_manager=log --webserver_interface=localhost --ipki_ca_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.0.57.254:37263 --webserver_interface=127.0.57.254 --webserver_port=42225 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_3.1557210500371863-231/raft_consensus-itest-cluster/master-0/logs --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_3.1557210500371863-231/raft_consensus-itest-cluster/master-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=true --rpc_reuseport=true with env {} WARNING: Logging before InitGoogleLogging() is written to STDERR W0507 06:32:45.699522 12247 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0507 06:32:45.699756 12247 flags.cc:404] Enabled unsafe flag: --never_fsync=true W0507 06:32:45.702194 12247 flags.cc:404] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true W0507 06:32:45.702260 12247 flags.cc:404] Enabled experimental flag: --ipki_ca_key_size=1024 W0507 06:32:45.702293 12247 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W0507 06:32:45.702328 12247 flags.cc:404] Enabled experimental flag: --tsk_num_rsa_bits=512 W0507 06:32:45.702368 12247 flags.cc:404] Enabled experimental flag: --rpc_reuseport=true I0507 06:32:45.707191 12247 master_main.cc:73] Master server non-default flags: --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_3.1557210500371863-231/raft_consensus-itest-cluster/master-0/data --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_3.1557210500371863-231/raft_consensus-itest-cluster/master-0/wal --ipki_ca_key_size=1024 --ipki_server_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.0.57.254:37263 --rpc_reuseport=true --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_3.1557210500371863-231/raft_consensus-itest-cluster/master-0/data/info.pb --webserver_interface=127.0.57.254 --webserver_port=42225 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_3.1557210500371863-231/raft_consensus-itest-cluster/master-0/logs --logbuflevel=-1 --logtostderr=true Master server version: kudu 1.10.0-cdh6.x-SNAPSHOT revision 7d0ee26697926e87af9ef395008799577f6ab43f build type FASTDEBUG built by jenkins at 06 May 2019 22:44:21 PST on quasar-knmpfx-1.vpc.cloudera.com build id 322017 ASAN enabled I0507 06:32:45.713759 12247 system_ntp.cc:202] NTP initialized. Skew: 500ppm Current error: 157016us I0507 06:32:45.715009 12247 fs_manager.cc:263] Metadata directory not provided I0507 06:32:45.715088 12247 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0507 06:32:45.722332 12247 fs_manager.cc:399] Time spent opening directory manager: real 0.005s user 0.002s sys 0.001s I0507 06:32:45.723076 12247 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0507 06:32:45.723667 12247 file_cache.cc:466] Constructed file cache lbm with capacity 419430 I0507 06:32:45.727166 12247 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s I0507 06:32:45.727272 12247 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_3.1557210500371863-231/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_3.1557210500371863-231/raft_consensus-itest-cluster/master-0/wal uuid: "e3246a64546142049e1d917a1380a9e1" format_stamp: "Formatted at 2019-05-07 06:32:44 on dist-test-slave-dist-test-slave-rqp6" I0507 06:32:45.728260 12247 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_3.1557210500371863-231/raft_consensus-itest-cluster/master-0/wal metadata directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_3.1557210500371863-231/raft_consensus-itest-cluster/master-0/wal 1 data directories: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_3.1557210500371863-231/raft_consensus-itest-cluster/master-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0507 06:32:45.752130 12247 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0507 06:32:45.767918 12247 rpc_server.cc:205] RPC server started. Bound to: 127.0.57.254:37263 I0507 06:32:45.769423 12247 webserver.cc:308] Webserver started at http://127.0.57.254:42225/ using document root and password file I0507 06:32:45.770563 12247 server_base.cc:608] Dumped server information to /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_3.1557210500371863-231/raft_consensus-itest-cluster/master-0/data/info.pb I0507 06:32:45.779734 12306 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P e3246a64546142049e1d917a1380a9e1: Bootstrap starting. I0507 06:32:45.780131 231 external_mini_cluster.cc:995] Started /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-master as pid 12247 I0507 06:32:45.780684 231 external_mini_cluster.cc:933] Running /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_3.1557210500371863-231/raft_consensus-itest-cluster/ts-0/wal --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_3.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data --block_manager=log --rpc_bind_addresses=127.0.57.193:39509 --local_ip_for_outbound_sockets=127.0.57.193 --webserver_port=46469 --webserver_interface=127.0.57.193 --tserver_master_addrs=127.0.57.254:37263 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_3.1557210500371863-231/raft_consensus-itest-cluster/ts-0/logs --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_3.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --heartbeat_incompatible_replica_management_is_fatal=true --heartbeat_interval_ms=500 --raft_prepare_replacement_before_eviction=true --raft_prepare_replacement_before_eviction=false with env {} I0507 06:32:45.795442 12306 log.cc:525] T 00000000000000000000000000000000 P e3246a64546142049e1d917a1380a9e1: Log is configured to *not* fsync() on all Append() calls I0507 06:32:45.806077 12306 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P e3246a64546142049e1d917a1380a9e1: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=4 ignored=0} mutations{seen=2 ignored=0} orphaned_commits=0. Pending: 0 replicates I0507 06:32:45.806550 12306 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P e3246a64546142049e1d917a1380a9e1: Bootstrap complete. I0507 06:32:45.816504 12306 raft_consensus.cc:340] T 00000000000000000000000000000000 P e3246a64546142049e1d917a1380a9e1 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3246a64546142049e1d917a1380a9e1" member_type: VOTER } I0507 06:32:45.817328 12306 raft_consensus.cc:705] T 00000000000000000000000000000000 P e3246a64546142049e1d917a1380a9e1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3246a64546142049e1d917a1380a9e1, State: Initialized, Role: FOLLOWER I0507 06:32:45.817953 12306 consensus_queue.cc:229] T 00000000000000000000000000000000 P e3246a64546142049e1d917a1380a9e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3246a64546142049e1d917a1380a9e1" member_type: VOTER } I0507 06:32:45.818137 12306 raft_consensus.cc:380] T 00000000000000000000000000000000 P e3246a64546142049e1d917a1380a9e1 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0507 06:32:45.818222 12306 raft_consensus.cc:472] T 00000000000000000000000000000000 P e3246a64546142049e1d917a1380a9e1 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0507 06:32:45.818327 12306 raft_consensus.cc:2886] T 00000000000000000000000000000000 P e3246a64546142049e1d917a1380a9e1 [term 1 FOLLOWER]: Advancing to term 2 I0507 06:32:45.822449 12306 raft_consensus.cc:494] T 00000000000000000000000000000000 P e3246a64546142049e1d917a1380a9e1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3246a64546142049e1d917a1380a9e1" member_type: VOTER } I0507 06:32:45.822947 12306 leader_election.cc:310] T 00000000000000000000000000000000 P e3246a64546142049e1d917a1380a9e1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e3246a64546142049e1d917a1380a9e1; no voters: I0507 06:32:45.824759 12306 leader_election.cc:296] T 00000000000000000000000000000000 P e3246a64546142049e1d917a1380a9e1 [CANDIDATE]: Term 2 election: Requested vote from peers I0507 06:32:45.825052 12309 raft_consensus.cc:2640] T 00000000000000000000000000000000 P e3246a64546142049e1d917a1380a9e1 [term 2 FOLLOWER]: Leader election won for term 2 I0507 06:32:45.826294 12309 raft_consensus.cc:667] T 00000000000000000000000000000000 P e3246a64546142049e1d917a1380a9e1 [term 2 LEADER]: Becoming Leader. State: Replica: e3246a64546142049e1d917a1380a9e1, State: Running, Role: LEADER I0507 06:32:45.826722 12309 consensus_queue.cc:206] T 00000000000000000000000000000000 P e3246a64546142049e1d917a1380a9e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3246a64546142049e1d917a1380a9e1" member_type: VOTER } I0507 06:32:45.828802 12310 sys_catalog.cc:337] T 00000000000000000000000000000000 P e3246a64546142049e1d917a1380a9e1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "e3246a64546142049e1d917a1380a9e1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3246a64546142049e1d917a1380a9e1" member_type: VOTER } } I0507 06:32:45.829370 12310 sys_catalog.cc:340] T 00000000000000000000000000000000 P e3246a64546142049e1d917a1380a9e1 [sys.catalog]: This master's current role is: LEADER I0507 06:32:45.828902 12311 sys_catalog.cc:337] T 00000000000000000000000000000000 P e3246a64546142049e1d917a1380a9e1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e3246a64546142049e1d917a1380a9e1. Latest consensus state: current_term: 2 leader_uuid: "e3246a64546142049e1d917a1380a9e1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3246a64546142049e1d917a1380a9e1" member_type: VOTER } } I0507 06:32:45.829100 12306 sys_catalog.cc:337] T 00000000000000000000000000000000 P e3246a64546142049e1d917a1380a9e1 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 2 leader_uuid: "e3246a64546142049e1d917a1380a9e1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3246a64546142049e1d917a1380a9e1" member_type: VOTER } } I0507 06:32:45.829887 12306 sys_catalog.cc:340] T 00000000000000000000000000000000 P e3246a64546142049e1d917a1380a9e1 [sys.catalog]: This master's current role is: LEADER I0507 06:32:45.831182 12306 sys_catalog.cc:424] T 00000000000000000000000000000000 P e3246a64546142049e1d917a1380a9e1 [sys.catalog]: configured and running, proceeding with master startup. I0507 06:32:45.829689 12311 sys_catalog.cc:340] T 00000000000000000000000000000000 P e3246a64546142049e1d917a1380a9e1 [sys.catalog]: This master's current role is: LEADER I0507 06:32:45.831358 12313 catalog_manager.cc:1055] Loading table and tablet metadata into memory... I0507 06:32:45.838958 12313 catalog_manager.cc:356] Loaded metadata for table TestTable [id=4efe84c6df5a44f4b051d1c093c35f57] I0507 06:32:45.840457 12313 catalog_manager.cc:410] Loaded metadata for tablet 9ad35469733c4e7cbeafc3da5cdbbbfc (table TestTable [id=4efe84c6df5a44f4b051d1c093c35f57]) I0507 06:32:45.844673 12313 catalog_manager.cc:1066] Initializing Kudu internal certificate authority... I0507 06:32:45.847421 12313 catalog_manager.cc:1075] Loading token signing keys... I0507 06:32:45.848343 12313 catalog_manager.cc:4303] T 00000000000000000000000000000000 P e3246a64546142049e1d917a1380a9e1: Loaded TSK: 0 WARNING: Logging before InitGoogleLogging() is written to STDERR W0507 06:32:45.953583 12307 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0507 06:32:45.953827 12307 flags.cc:404] Enabled unsafe flag: --never_fsync=true W0507 06:32:45.955932 12307 flags.cc:404] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false W0507 06:32:45.955992 12307 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W0507 06:32:45.956092 12307 flags.cc:404] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.57.193 I0507 06:32:45.964711 12307 tablet_server_main.cc:78] Tablet server non-default flags: --raft_prepare_replacement_before_eviction=false --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_3.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_3.1557210500371863-231/raft_consensus-itest-cluster/ts-0/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.0.57.193:39509 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_3.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data/info.pb --webserver_interface=127.0.57.193 --webserver_port=46469 --heartbeat_interval_ms=500 --tserver_master_addrs=127.0.57.254:37263 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.0.57.193 --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_3.1557210500371863-231/raft_consensus-itest-cluster/ts-0/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.10.0-cdh6.x-SNAPSHOT revision 7d0ee26697926e87af9ef395008799577f6ab43f build type FASTDEBUG built by jenkins at 06 May 2019 22:44:21 PST on quasar-knmpfx-1.vpc.cloudera.com build id 322017 ASAN enabled I0507 06:32:45.973649 12307 system_ntp.cc:202] NTP initialized. Skew: 500ppm Current error: 157016us I0507 06:32:45.974440 12307 fs_manager.cc:263] Metadata directory not provided I0507 06:32:45.974521 12307 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0507 06:32:45.983419 12307 fs_manager.cc:399] Time spent opening directory manager: real 0.007s user 0.004s sys 0.004s I0507 06:32:45.984172 12307 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0507 06:32:45.984885 12307 file_cache.cc:466] Constructed file cache lbm with capacity 419430 I0507 06:32:45.998919 12307 fs_manager.cc:419] Time spent opening block manager: real 0.013s user 0.000s sys 0.011s I0507 06:32:45.999050 12307 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_3.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_3.1557210500371863-231/raft_consensus-itest-cluster/ts-0/wal uuid: "a82d40cf7c44462e88ae97100184ca36" format_stamp: "Formatted at 2019-05-07 06:32:45 on dist-test-slave-dist-test-slave-rqp6" I0507 06:32:46.000270 12307 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_3.1557210500371863-231/raft_consensus-itest-cluster/ts-0/wal metadata directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_3.1557210500371863-231/raft_consensus-itest-cluster/ts-0/wal 1 data directories: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_3.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0507 06:32:46.053964 12307 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0507 06:32:46.056002 12307 ts_tablet_manager.cc:344] Loading tablet metadata (0/1 complete) I0507 06:32:46.060282 12307 ts_tablet_manager.cc:356] Loaded tablet metadata (1 total tablets, 1 live tablets) I0507 06:32:46.060343 12307 ts_tablet_manager.cc:362] Registering tablets (0/1 complete) I0507 06:32:46.064723 12307 ts_tablet_manager.cc:376] Registered 1 tablets I0507 06:32:46.065439 12331 tablet_bootstrap.cc:439] T 9ad35469733c4e7cbeafc3da5cdbbbfc P a82d40cf7c44462e88ae97100184ca36: Bootstrap starting. I0507 06:32:46.092634 12331 log.cc:525] T 9ad35469733c4e7cbeafc3da5cdbbbfc P a82d40cf7c44462e88ae97100184ca36: Log is configured to *not* fsync() on all Append() calls I0507 06:32:46.097085 12331 tablet_bootstrap.cc:439] T 9ad35469733c4e7cbeafc3da5cdbbbfc P a82d40cf7c44462e88ae97100184ca36: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I0507 06:32:46.098006 12331 tablet_bootstrap.cc:439] T 9ad35469733c4e7cbeafc3da5cdbbbfc P a82d40cf7c44462e88ae97100184ca36: Bootstrap complete. I0507 06:32:46.099107 12331 ts_tablet_manager.cc:1053] T 9ad35469733c4e7cbeafc3da5cdbbbfc P a82d40cf7c44462e88ae97100184ca36: Time spent bootstrapping tablet: real 0.034s user 0.008s sys 0.013s I0507 06:32:46.110028 12331 raft_consensus.cc:340] T 9ad35469733c4e7cbeafc3da5cdbbbfc P a82d40cf7c44462e88ae97100184ca36 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a82d40cf7c44462e88ae97100184ca36" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 39509 } } I0507 06:32:46.112900 12331 raft_consensus.cc:705] T 9ad35469733c4e7cbeafc3da5cdbbbfc P a82d40cf7c44462e88ae97100184ca36 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a82d40cf7c44462e88ae97100184ca36, State: Initialized, Role: FOLLOWER I0507 06:32:46.113495 12331 consensus_queue.cc:229] T 9ad35469733c4e7cbeafc3da5cdbbbfc P a82d40cf7c44462e88ae97100184ca36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a82d40cf7c44462e88ae97100184ca36" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 39509 } } I0507 06:32:46.113771 12331 raft_consensus.cc:380] T 9ad35469733c4e7cbeafc3da5cdbbbfc P a82d40cf7c44462e88ae97100184ca36 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0507 06:32:46.113865 12331 raft_consensus.cc:472] T 9ad35469733c4e7cbeafc3da5cdbbbfc P a82d40cf7c44462e88ae97100184ca36 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0507 06:32:46.114006 12331 raft_consensus.cc:2886] T 9ad35469733c4e7cbeafc3da5cdbbbfc P a82d40cf7c44462e88ae97100184ca36 [term 1 FOLLOWER]: Advancing to term 2 I0507 06:32:46.117301 12331 raft_consensus.cc:494] T 9ad35469733c4e7cbeafc3da5cdbbbfc P a82d40cf7c44462e88ae97100184ca36 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a82d40cf7c44462e88ae97100184ca36" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 39509 } } I0507 06:32:46.117825 12331 leader_election.cc:310] T 9ad35469733c4e7cbeafc3da5cdbbbfc P a82d40cf7c44462e88ae97100184ca36 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a82d40cf7c44462e88ae97100184ca36; no voters: I0507 06:32:46.121124 12331 leader_election.cc:296] T 9ad35469733c4e7cbeafc3da5cdbbbfc P a82d40cf7c44462e88ae97100184ca36 [CANDIDATE]: Term 2 election: Requested vote from peers I0507 06:32:46.121286 12411 raft_consensus.cc:2640] T 9ad35469733c4e7cbeafc3da5cdbbbfc P a82d40cf7c44462e88ae97100184ca36 [term 2 FOLLOWER]: Leader election won for term 2 I0507 06:32:46.125528 12411 raft_consensus.cc:667] T 9ad35469733c4e7cbeafc3da5cdbbbfc P a82d40cf7c44462e88ae97100184ca36 [term 2 LEADER]: Becoming Leader. State: Replica: a82d40cf7c44462e88ae97100184ca36, State: Running, Role: LEADER I0507 06:32:46.125991 12331 ts_tablet_manager.cc:1081] T 9ad35469733c4e7cbeafc3da5cdbbbfc P a82d40cf7c44462e88ae97100184ca36: Time spent starting tablet: real 0.027s user 0.013s sys 0.012s I0507 06:32:46.126484 12411 consensus_queue.cc:206] T 9ad35469733c4e7cbeafc3da5cdbbbfc P a82d40cf7c44462e88ae97100184ca36 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a82d40cf7c44462e88ae97100184ca36" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 39509 } } I0507 06:32:46.133849 12307 rpc_server.cc:205] RPC server started. Bound to: 127.0.57.193:39509 I0507 06:32:46.135077 12307 webserver.cc:308] Webserver started at http://127.0.57.193:46469/ using document root and password file I0507 06:32:46.135737 12307 server_base.cc:608] Dumped server information to /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_3.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data/info.pb I0507 06:32:46.144357 231 external_mini_cluster.cc:995] Started /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver as pid 12307 I0507 06:32:46.149559 12442 heartbeater.cc:345] Connected to a master server at 127.0.57.254:37263 I0507 06:32:46.149793 12442 heartbeater.cc:426] Registering TS with master... I0507 06:32:46.150275 12442 heartbeater.cc:475] Master 127.0.57.254:37263 requested a full tablet report, sending... W0507 06:32:46.152433 12273 master_service.cc:205] Configuration error: replica replacement scheme (EVICT_FIRST) of the tablet server a82d40cf7c44462e88ae97100184ca36 at 127.0.57.193:43257 differs from the catalog manager's (PREPARE_REPLACEMENT_BEFORE_EVICTION); they must be run with the same scheme (controlled by the --raft_prepare_replacement_before_eviction flag) W0507 06:32:46.153548 12442 heartbeater.cc:587] Failed to heartbeat to 127.0.57.254:37263 (0 consecutive failures): Configuration error: replica replacement scheme (EVICT_FIRST) of the tablet server a82d40cf7c44462e88ae97100184ca36 at 127.0.57.193:43257 differs from the catalog manager's (PREPARE_REPLACEMENT_BEFORE_EVICTION); they must be run with the same scheme (controlled by the --raft_prepare_replacement_before_eviction flag) F0507 06:32:46.153609 12442 heartbeater.cc:620] master detected incompatibility: Configuration error: replica replacement scheme (EVICT_FIRST) of the tablet server a82d40cf7c44462e88ae97100184ca36 at 127.0.57.193:43257 differs from the catalog manager's (PREPARE_REPLACEMENT_BEFORE_EVICTION); they must be run with the same scheme (controlled by the --raft_prepare_replacement_before_eviction flag) *** Check failure stack trace: *** @ 0x7f773763531d google::LogMessage::Fail() at ??:0 @ 0x7f77376371dd google::LogMessage::SendToLog() at ??:0 @ 0x7f7737634e59 google::LogMessage::Flush() at ??:0 @ 0x7f7737637c7f google::LogMessageFatal::~LogMessageFatal() at ??:0 @ 0x7f774554646b kudu::tserver::Heartbeater::Thread::RunThread() at ??:0 @ 0x7f7745559b83 boost::_bi::bind_t<>::operator()() at ??:0 @ 0x7f773b7e1d2c boost::function0<>::operator()() at ??:0 @ 0x7f7738890c6b kudu::Thread::SuperviseThread() at ??:0 @ 0x7f773ec30184 start_thread at ??:0 @ 0x7f7733c64ffd clone at ??:0 I0507 06:32:47.862821 231 external_mini_cluster.cc:933] Running /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_3.1557210500371863-231/raft_consensus-itest-cluster/ts-0/wal --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_3.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data --block_manager=log --rpc_bind_addresses=127.0.57.193:39509 --local_ip_for_outbound_sockets=127.0.57.193 --webserver_port=46469 --webserver_interface=127.0.57.193 --tserver_master_addrs=127.0.57.254:37263 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_3.1557210500371863-231/raft_consensus-itest-cluster/ts-0/logs --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_3.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --heartbeat_incompatible_replica_management_is_fatal=true --heartbeat_interval_ms=500 --raft_prepare_replacement_before_eviction=true --heartbeat_inject_required_feature_flag=999 with env {} WARNING: Logging before InitGoogleLogging() is written to STDERR W0507 06:32:48.031126 12458 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0507 06:32:48.031337 12458 flags.cc:404] Enabled unsafe flag: --heartbeat_inject_required_feature_flag=999 W0507 06:32:48.031378 12458 flags.cc:404] Enabled unsafe flag: --never_fsync=true W0507 06:32:48.033437 12458 flags.cc:404] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true W0507 06:32:48.033505 12458 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W0507 06:32:48.033617 12458 flags.cc:404] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.57.193 I0507 06:32:48.037540 12458 tablet_server_main.cc:78] Tablet server non-default flags: --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_3.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_3.1557210500371863-231/raft_consensus-itest-cluster/ts-0/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.0.57.193:39509 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_3.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data/info.pb --webserver_interface=127.0.57.193 --webserver_port=46469 --heartbeat_inject_required_feature_flag=999 --heartbeat_interval_ms=500 --tserver_master_addrs=127.0.57.254:37263 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.0.57.193 --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_3.1557210500371863-231/raft_consensus-itest-cluster/ts-0/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.10.0-cdh6.x-SNAPSHOT revision 7d0ee26697926e87af9ef395008799577f6ab43f build type FASTDEBUG built by jenkins at 06 May 2019 22:44:21 PST on quasar-knmpfx-1.vpc.cloudera.com build id 322017 ASAN enabled I0507 06:32:48.044904 12458 system_ntp.cc:202] NTP initialized. Skew: 500ppm Current error: 158516us I0507 06:32:48.045558 12458 fs_manager.cc:263] Metadata directory not provided I0507 06:32:48.045667 12458 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0507 06:32:48.051498 12458 fs_manager.cc:399] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s I0507 06:32:48.052175 12458 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0507 06:32:48.052758 12458 file_cache.cc:466] Constructed file cache lbm with capacity 419430 I0507 06:32:48.055657 12458 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s I0507 06:32:48.055730 12458 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_3.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_3.1557210500371863-231/raft_consensus-itest-cluster/ts-0/wal uuid: "a82d40cf7c44462e88ae97100184ca36" format_stamp: "Formatted at 2019-05-07 06:32:45 on dist-test-slave-dist-test-slave-rqp6" I0507 06:32:48.056756 12458 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_3.1557210500371863-231/raft_consensus-itest-cluster/ts-0/wal metadata directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_3.1557210500371863-231/raft_consensus-itest-cluster/ts-0/wal 1 data directories: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_3.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0507 06:32:48.102924 12458 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0507 06:32:48.104801 12458 ts_tablet_manager.cc:344] Loading tablet metadata (0/1 complete) I0507 06:32:48.108912 12458 ts_tablet_manager.cc:356] Loaded tablet metadata (1 total tablets, 1 live tablets) I0507 06:32:48.108981 12458 ts_tablet_manager.cc:362] Registering tablets (0/1 complete) I0507 06:32:48.111775 12458 ts_tablet_manager.cc:376] Registered 1 tablets I0507 06:32:48.112246 12473 tablet_bootstrap.cc:439] T 9ad35469733c4e7cbeafc3da5cdbbbfc P a82d40cf7c44462e88ae97100184ca36: Bootstrap starting. I0507 06:32:48.125376 12473 log.cc:525] T 9ad35469733c4e7cbeafc3da5cdbbbfc P a82d40cf7c44462e88ae97100184ca36: Log is configured to *not* fsync() on all Append() calls I0507 06:32:48.132182 12473 tablet_bootstrap.cc:439] T 9ad35469733c4e7cbeafc3da5cdbbbfc P a82d40cf7c44462e88ae97100184ca36: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I0507 06:32:48.132704 12473 tablet_bootstrap.cc:439] T 9ad35469733c4e7cbeafc3da5cdbbbfc P a82d40cf7c44462e88ae97100184ca36: Bootstrap complete. I0507 06:32:48.133778 12473 ts_tablet_manager.cc:1053] T 9ad35469733c4e7cbeafc3da5cdbbbfc P a82d40cf7c44462e88ae97100184ca36: Time spent bootstrapping tablet: real 0.022s user 0.013s sys 0.007s I0507 06:32:48.143625 12473 raft_consensus.cc:340] T 9ad35469733c4e7cbeafc3da5cdbbbfc P a82d40cf7c44462e88ae97100184ca36 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a82d40cf7c44462e88ae97100184ca36" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 39509 } } I0507 06:32:48.146245 12473 raft_consensus.cc:705] T 9ad35469733c4e7cbeafc3da5cdbbbfc P a82d40cf7c44462e88ae97100184ca36 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: a82d40cf7c44462e88ae97100184ca36, State: Initialized, Role: FOLLOWER I0507 06:32:48.147653 12473 consensus_queue.cc:229] T 9ad35469733c4e7cbeafc3da5cdbbbfc P a82d40cf7c44462e88ae97100184ca36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a82d40cf7c44462e88ae97100184ca36" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 39509 } } I0507 06:32:48.147986 12473 raft_consensus.cc:380] T 9ad35469733c4e7cbeafc3da5cdbbbfc P a82d40cf7c44462e88ae97100184ca36 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0507 06:32:48.148249 12473 raft_consensus.cc:472] T 9ad35469733c4e7cbeafc3da5cdbbbfc P a82d40cf7c44462e88ae97100184ca36 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0507 06:32:48.148586 12473 raft_consensus.cc:2886] T 9ad35469733c4e7cbeafc3da5cdbbbfc P a82d40cf7c44462e88ae97100184ca36 [term 2 FOLLOWER]: Advancing to term 3 I0507 06:32:48.153533 12473 raft_consensus.cc:494] T 9ad35469733c4e7cbeafc3da5cdbbbfc P a82d40cf7c44462e88ae97100184ca36 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a82d40cf7c44462e88ae97100184ca36" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 39509 } } I0507 06:32:48.154294 12473 leader_election.cc:310] T 9ad35469733c4e7cbeafc3da5cdbbbfc P a82d40cf7c44462e88ae97100184ca36 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a82d40cf7c44462e88ae97100184ca36; no voters: I0507 06:32:48.155391 12473 leader_election.cc:296] T 9ad35469733c4e7cbeafc3da5cdbbbfc P a82d40cf7c44462e88ae97100184ca36 [CANDIDATE]: Term 3 election: Requested vote from peers I0507 06:32:48.155800 12506 raft_consensus.cc:2640] T 9ad35469733c4e7cbeafc3da5cdbbbfc P a82d40cf7c44462e88ae97100184ca36 [term 3 FOLLOWER]: Leader election won for term 3 I0507 06:32:48.157214 12473 ts_tablet_manager.cc:1081] T 9ad35469733c4e7cbeafc3da5cdbbbfc P a82d40cf7c44462e88ae97100184ca36: Time spent starting tablet: real 0.023s user 0.018s sys 0.005s I0507 06:32:48.157346 12506 raft_consensus.cc:667] T 9ad35469733c4e7cbeafc3da5cdbbbfc P a82d40cf7c44462e88ae97100184ca36 [term 3 LEADER]: Becoming Leader. State: Replica: a82d40cf7c44462e88ae97100184ca36, State: Running, Role: LEADER I0507 06:32:48.158149 12506 consensus_queue.cc:206] T 9ad35469733c4e7cbeafc3da5cdbbbfc P a82d40cf7c44462e88ae97100184ca36 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a82d40cf7c44462e88ae97100184ca36" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 39509 } } I0507 06:32:48.177088 12458 rpc_server.cc:205] RPC server started. Bound to: 127.0.57.193:39509 I0507 06:32:48.178373 12458 webserver.cc:308] Webserver started at http://127.0.57.193:46469/ using document root and password file I0507 06:32:48.179098 12458 server_base.cc:608] Dumped server information to /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig_3.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data/info.pb I0507 06:32:48.187451 231 external_mini_cluster.cc:995] Started /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver as pid 12458 I0507 06:32:48.188331 12584 heartbeater.cc:345] Connected to a master server at 127.0.57.254:37263 I0507 06:32:48.188558 12584 heartbeater.cc:426] Registering TS with master... I0507 06:32:48.189110 12584 heartbeater.cc:475] Master 127.0.57.254:37263 requested a full tablet report, sending... W0507 06:32:48.190606 12254 messenger.cc:389] Unable to handle RPC call: Not implemented: call requires unsupported application feature flags: 999 W0507 06:32:48.191275 12584 heartbeater.cc:587] Failed to heartbeat to 127.0.57.254:37263 (0 consecutive failures): Remote error: Failed to send heartbeat to master: unsupported feature flags F0507 06:32:48.191397 12584 heartbeater.cc:620] master does not support required feature flags: Remote error: Failed to send heartbeat to master: unsupported feature flags *** Check failure stack trace: *** @ 0x7fb21de1431d google::LogMessage::Fail() at ??:0 @ 0x7fb21de161dd google::LogMessage::SendToLog() at ??:0 @ 0x7fb21de13e59 google::LogMessage::Flush() at ??:0 @ 0x7fb21de16c7f google::LogMessageFatal::~LogMessageFatal() at ??:0 @ 0x7fb22bd2546b kudu::tserver::Heartbeater::Thread::RunThread() at ??:0 @ 0x7fb22bd38b83 boost::_bi::bind_t<>::operator()() at ??:0 @ 0x7fb221fc0d2c boost::function0<>::operator()() at ??:0 @ 0x7fb21f06fc6b kudu::Thread::SuperviseThread() at ??:0 @ 0x7fb22540f184 start_thread at ??:0 @ 0x7fb21a443ffd clone at ??:0 I0507 06:32:49.901108 231 external_mini_cluster.cc:1172] Attempting to check leaks for /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-master pid 12247 I0507 06:32:49.902045 12303 generic_service.cc:171] Checking for leaks (request via RPC) I0507 06:32:49.971220 231 external_mini_cluster.cc:1131] Killing /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-master with pid 12247 [ OK ] IncompatibleReplicaReplacementSchemesITest.MasterAndTserverMisconfig/3 (5424 ms) [----------] 4 tests from IncompatibleReplicaReplacementSchemesITest (21284 ms total) [----------] 6 tests from ReplicaBehindWalGcThresholdITest [ RUN ] ReplicaBehindWalGcThresholdITest.ReplicaReplacement/0 I0507 06:32:49.993176 231 test_util.cc:212] Using random seed: 1657351622 I0507 06:32:49.996021 231 ts_itest-base.cc:112] Starting cluster with: I0507 06:32:49.996068 231 ts_itest-base.cc:113] -------------- I0507 06:32:49.996100 231 ts_itest-base.cc:114] 3 tablet servers I0507 06:32:49.996126 231 ts_itest-base.cc:115] 3 replicas per TS I0507 06:32:49.996163 231 ts_itest-base.cc:116] -------------- I0507 06:32:49.997745 231 external_mini_cluster.cc:933] Running /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-master /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-master --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_0.1557210500371863-231/raft_consensus-itest-cluster/master-0/wal --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_0.1557210500371863-231/raft_consensus-itest-cluster/master-0/data --block_manager=log --webserver_interface=localhost --ipki_ca_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.0.57.254:36417 --webserver_interface=127.0.57.254 --webserver_port=0 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_0.1557210500371863-231/raft_consensus-itest-cluster/master-0/logs --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_0.1557210500371863-231/raft_consensus-itest-cluster/master-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=true --master_add_server_when_underreplicated=false --rpc_reuseport=true with env {} WARNING: Logging before InitGoogleLogging() is written to STDERR W0507 06:32:50.211149 12603 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0507 06:32:50.211506 12603 flags.cc:404] Enabled unsafe flag: --never_fsync=true W0507 06:32:50.214241 12603 flags.cc:404] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true W0507 06:32:50.214310 12603 flags.cc:404] Enabled experimental flag: --ipki_ca_key_size=1024 W0507 06:32:50.214341 12603 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W0507 06:32:50.214367 12603 flags.cc:404] Enabled experimental flag: --tsk_num_rsa_bits=512 W0507 06:32:50.214414 12603 flags.cc:404] Enabled experimental flag: --rpc_reuseport=true I0507 06:32:50.219357 12603 master_main.cc:73] Master server non-default flags: --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_0.1557210500371863-231/raft_consensus-itest-cluster/master-0/data --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_0.1557210500371863-231/raft_consensus-itest-cluster/master-0/wal --master_add_server_when_underreplicated=false --ipki_ca_key_size=1024 --ipki_server_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.0.57.254:36417 --rpc_reuseport=true --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_0.1557210500371863-231/raft_consensus-itest-cluster/master-0/data/info.pb --webserver_interface=127.0.57.254 --webserver_port=0 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_0.1557210500371863-231/raft_consensus-itest-cluster/master-0/logs --logbuflevel=-1 --logtostderr=true Master server version: kudu 1.10.0-cdh6.x-SNAPSHOT revision 7d0ee26697926e87af9ef395008799577f6ab43f build type FASTDEBUG built by jenkins at 06 May 2019 22:44:21 PST on quasar-knmpfx-1.vpc.cloudera.com build id 322017 ASAN enabled I0507 06:32:50.226742 12603 system_ntp.cc:202] NTP initialized. Skew: 500ppm Current error: 159516us I0507 06:32:50.228127 12603 fs_manager.cc:263] Metadata directory not provided I0507 06:32:50.228196 12603 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0507 06:32:50.228485 12603 server_base.cc:433] Could not load existing FS layout: Not found: could not find a healthy instance file I0507 06:32:50.228600 12603 server_base.cc:434] Attempting to create new FS layout instead I0507 06:32:50.231292 12603 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_0.1557210500371863-231/raft_consensus-itest-cluster/master-0/data/instance: uuid: "a1bfe1b2ebf24d0ca70399022578708a" format_stamp: "Formatted at 2019-05-07 06:32:50 on dist-test-slave-dist-test-slave-rqp6" I0507 06:32:50.231843 12603 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_0.1557210500371863-231/raft_consensus-itest-cluster/master-0/wal/instance: uuid: "a1bfe1b2ebf24d0ca70399022578708a" format_stamp: "Formatted at 2019-05-07 06:32:50 on dist-test-slave-dist-test-slave-rqp6" I0507 06:32:50.239750 12603 fs_manager.cc:503] Time spent creating directory manager: real 0.008s user 0.009s sys 0.000s I0507 06:32:50.241041 12603 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0507 06:32:50.241672 12603 file_cache.cc:466] Constructed file cache lbm with capacity 419430 I0507 06:32:50.245710 12603 fs_manager.cc:419] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s I0507 06:32:50.245810 12603 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_0.1557210500371863-231/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_0.1557210500371863-231/raft_consensus-itest-cluster/master-0/wal uuid: "a1bfe1b2ebf24d0ca70399022578708a" format_stamp: "Formatted at 2019-05-07 06:32:50 on dist-test-slave-dist-test-slave-rqp6" I0507 06:32:50.245985 12603 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_0.1557210500371863-231/raft_consensus-itest-cluster/master-0/wal metadata directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_0.1557210500371863-231/raft_consensus-itest-cluster/master-0/wal 1 data directories: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_0.1557210500371863-231/raft_consensus-itest-cluster/master-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0507 06:32:50.278942 12603 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0507 06:32:50.294711 12603 rpc_server.cc:205] RPC server started. Bound to: 127.0.57.254:36417 I0507 06:32:50.296192 12603 webserver.cc:308] Webserver started at http://127.0.57.254:32787/ using document root and password file I0507 06:32:50.297125 12603 server_base.cc:608] Dumped server information to /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_0.1557210500371863-231/raft_consensus-itest-cluster/master-0/data/info.pb I0507 06:32:50.300856 12662 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0507 06:32:50.301131 231 external_mini_cluster.cc:995] Started /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-master as pid 12603 I0507 06:32:50.302011 231 external_mini_cluster.cc:933] Running /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_0.1557210500371863-231/raft_consensus-itest-cluster/ts-0/wal --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_0.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data --block_manager=log --rpc_bind_addresses=127.0.57.193:0 --local_ip_for_outbound_sockets=127.0.57.193 --webserver_interface=127.0.57.193 --webserver_port=0 --tserver_master_addrs=127.0.57.254:36417 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_0.1557210500371863-231/raft_consensus-itest-cluster/ts-0/logs --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_0.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=true --follower_unavailable_considered_failed_sec=5 --log_compression_codec=none --log_cache_size_limit_mb=1 --log_segment_size_mb=1 --log_async_preallocate_segments=false --log_min_segments_to_retain=1 --log_max_segments_to_retain=3 --maintenance_manager_polling_interval_ms=100 --log_target_replay_size_mb=1 --max_cell_size_bytes=1000000 with env {} I0507 06:32:50.313738 12662 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P a1bfe1b2ebf24d0ca70399022578708a: Bootstrap starting. I0507 06:32:50.316419 12662 tablet_bootstrap.cc:590] T 00000000000000000000000000000000 P a1bfe1b2ebf24d0ca70399022578708a: No blocks or log segments found. Creating new log. I0507 06:32:50.321629 12662 log.cc:525] T 00000000000000000000000000000000 P a1bfe1b2ebf24d0ca70399022578708a: Log is configured to *not* fsync() on all Append() calls I0507 06:32:50.330487 12662 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P a1bfe1b2ebf24d0ca70399022578708a: No bootstrap required, opened a new log I0507 06:32:50.340209 12662 raft_consensus.cc:340] T 00000000000000000000000000000000 P a1bfe1b2ebf24d0ca70399022578708a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1bfe1b2ebf24d0ca70399022578708a" member_type: VOTER } I0507 06:32:50.340544 12662 raft_consensus.cc:366] T 00000000000000000000000000000000 P a1bfe1b2ebf24d0ca70399022578708a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0507 06:32:50.340654 12662 raft_consensus.cc:705] T 00000000000000000000000000000000 P a1bfe1b2ebf24d0ca70399022578708a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1bfe1b2ebf24d0ca70399022578708a, State: Initialized, Role: FOLLOWER I0507 06:32:50.341279 12662 consensus_queue.cc:229] T 00000000000000000000000000000000 P a1bfe1b2ebf24d0ca70399022578708a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1bfe1b2ebf24d0ca70399022578708a" member_type: VOTER } I0507 06:32:50.341504 12662 raft_consensus.cc:380] T 00000000000000000000000000000000 P a1bfe1b2ebf24d0ca70399022578708a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0507 06:32:50.341596 12662 raft_consensus.cc:472] T 00000000000000000000000000000000 P a1bfe1b2ebf24d0ca70399022578708a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0507 06:32:50.341703 12662 raft_consensus.cc:2886] T 00000000000000000000000000000000 P a1bfe1b2ebf24d0ca70399022578708a [term 0 FOLLOWER]: Advancing to term 1 I0507 06:32:50.344110 12662 raft_consensus.cc:494] T 00000000000000000000000000000000 P a1bfe1b2ebf24d0ca70399022578708a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1bfe1b2ebf24d0ca70399022578708a" member_type: VOTER } I0507 06:32:50.344712 12662 leader_election.cc:310] T 00000000000000000000000000000000 P a1bfe1b2ebf24d0ca70399022578708a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a1bfe1b2ebf24d0ca70399022578708a; no voters: I0507 06:32:50.348742 12662 leader_election.cc:296] T 00000000000000000000000000000000 P a1bfe1b2ebf24d0ca70399022578708a [CANDIDATE]: Term 1 election: Requested vote from peers I0507 06:32:50.348928 12665 raft_consensus.cc:2640] T 00000000000000000000000000000000 P a1bfe1b2ebf24d0ca70399022578708a [term 1 FOLLOWER]: Leader election won for term 1 I0507 06:32:50.349795 12665 raft_consensus.cc:667] T 00000000000000000000000000000000 P a1bfe1b2ebf24d0ca70399022578708a [term 1 LEADER]: Becoming Leader. State: Replica: a1bfe1b2ebf24d0ca70399022578708a, State: Running, Role: LEADER I0507 06:32:50.350425 12665 consensus_queue.cc:206] T 00000000000000000000000000000000 P a1bfe1b2ebf24d0ca70399022578708a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1bfe1b2ebf24d0ca70399022578708a" member_type: VOTER } I0507 06:32:50.352463 12665 sys_catalog.cc:337] T 00000000000000000000000000000000 P a1bfe1b2ebf24d0ca70399022578708a [sys.catalog]: SysCatalogTable state changed. Reason: New leader a1bfe1b2ebf24d0ca70399022578708a. Latest consensus state: current_term: 1 leader_uuid: "a1bfe1b2ebf24d0ca70399022578708a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1bfe1b2ebf24d0ca70399022578708a" member_type: VOTER } } I0507 06:32:50.352959 12665 sys_catalog.cc:340] T 00000000000000000000000000000000 P a1bfe1b2ebf24d0ca70399022578708a [sys.catalog]: This master's current role is: LEADER I0507 06:32:50.353579 12662 sys_catalog.cc:337] T 00000000000000000000000000000000 P a1bfe1b2ebf24d0ca70399022578708a [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 1 leader_uuid: "a1bfe1b2ebf24d0ca70399022578708a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1bfe1b2ebf24d0ca70399022578708a" member_type: VOTER } } I0507 06:32:50.353278 12667 sys_catalog.cc:337] T 00000000000000000000000000000000 P a1bfe1b2ebf24d0ca70399022578708a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a1bfe1b2ebf24d0ca70399022578708a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1bfe1b2ebf24d0ca70399022578708a" member_type: VOTER } } I0507 06:32:50.354476 12662 sys_catalog.cc:340] T 00000000000000000000000000000000 P a1bfe1b2ebf24d0ca70399022578708a [sys.catalog]: This master's current role is: LEADER I0507 06:32:50.355105 12667 sys_catalog.cc:340] T 00000000000000000000000000000000 P a1bfe1b2ebf24d0ca70399022578708a [sys.catalog]: This master's current role is: LEADER I0507 06:32:50.357216 12669 catalog_manager.cc:1055] Loading table and tablet metadata into memory... I0507 06:32:50.358983 12662 sys_catalog.cc:424] T 00000000000000000000000000000000 P a1bfe1b2ebf24d0ca70399022578708a [sys.catalog]: configured and running, proceeding with master startup. I0507 06:32:50.362068 12669 catalog_manager.cc:1066] Initializing Kudu internal certificate authority... I0507 06:32:50.434878 12669 catalog_manager.cc:950] Generated new certificate authority record I0507 06:32:50.436661 12669 catalog_manager.cc:1075] Loading token signing keys... I0507 06:32:50.447685 12669 catalog_manager.cc:4292] T 00000000000000000000000000000000 P a1bfe1b2ebf24d0ca70399022578708a: Generated new TSK 0 WARNING: Logging before InitGoogleLogging() is written to STDERR W0507 06:32:50.480319 12663 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0507 06:32:50.480626 12663 flags.cc:404] Enabled unsafe flag: --max_cell_size_bytes=1000000 W0507 06:32:50.480690 12663 flags.cc:404] Enabled unsafe flag: --never_fsync=true W0507 06:32:50.482766 12663 flags.cc:404] Enabled experimental flag: --log_compression_codec=none W0507 06:32:50.482827 12663 flags.cc:404] Enabled experimental flag: --log_max_segments_to_retain=3 W0507 06:32:50.482873 12663 flags.cc:404] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true W0507 06:32:50.482913 12663 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W0507 06:32:50.482990 12663 flags.cc:404] Enabled experimental flag: --log_target_replay_size_mb=1 W0507 06:32:50.483042 12663 flags.cc:404] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.57.193 I0507 06:32:50.496709 12663 tablet_server_main.cc:78] Tablet server non-default flags: --follower_unavailable_considered_failed_sec=5 --log_compression_codec=none --log_max_segments_to_retain=3 --log_cache_size_limit_mb=1 --log_async_preallocate_segments=false --log_segment_size_mb=1 --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_0.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_0.1557210500371863-231/raft_consensus-itest-cluster/ts-0/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.0.57.193:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_0.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data/info.pb --webserver_interface=127.0.57.193 --webserver_port=0 --max_cell_size_bytes=1000000 --tserver_master_addrs=127.0.57.254:36417 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --log_target_replay_size_mb=1 --maintenance_manager_polling_interval_ms=100 --enable_minidumps=false --local_ip_for_outbound_sockets=127.0.57.193 --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_0.1557210500371863-231/raft_consensus-itest-cluster/ts-0/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.10.0-cdh6.x-SNAPSHOT revision 7d0ee26697926e87af9ef395008799577f6ab43f build type FASTDEBUG built by jenkins at 06 May 2019 22:44:21 PST on quasar-knmpfx-1.vpc.cloudera.com build id 322017 ASAN enabled I0507 06:32:50.504421 12663 system_ntp.cc:202] NTP initialized. Skew: 500ppm Current error: 159516us I0507 06:32:50.505118 12663 fs_manager.cc:263] Metadata directory not provided I0507 06:32:50.505178 12663 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0507 06:32:50.505513 12663 server_base.cc:433] Could not load existing FS layout: Not found: could not find a healthy instance file I0507 06:32:50.505575 12663 server_base.cc:434] Attempting to create new FS layout instead I0507 06:32:50.507928 12663 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_0.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data/instance: uuid: "e167493e37494c2896e3b5fe35a5a30d" format_stamp: "Formatted at 2019-05-07 06:32:50 on dist-test-slave-dist-test-slave-rqp6" I0507 06:32:50.508517 12663 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_0.1557210500371863-231/raft_consensus-itest-cluster/ts-0/wal/instance: uuid: "e167493e37494c2896e3b5fe35a5a30d" format_stamp: "Formatted at 2019-05-07 06:32:50 on dist-test-slave-dist-test-slave-rqp6" I0507 06:32:50.519552 12663 fs_manager.cc:503] Time spent creating directory manager: real 0.011s user 0.005s sys 0.004s I0507 06:32:50.520809 12663 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0507 06:32:50.521430 12663 file_cache.cc:466] Constructed file cache lbm with capacity 419430 I0507 06:32:50.524899 12663 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s I0507 06:32:50.524993 12663 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_0.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_0.1557210500371863-231/raft_consensus-itest-cluster/ts-0/wal uuid: "e167493e37494c2896e3b5fe35a5a30d" format_stamp: "Formatted at 2019-05-07 06:32:50 on dist-test-slave-dist-test-slave-rqp6" I0507 06:32:50.525156 12663 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_0.1557210500371863-231/raft_consensus-itest-cluster/ts-0/wal metadata directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_0.1557210500371863-231/raft_consensus-itest-cluster/ts-0/wal 1 data directories: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_0.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0507 06:32:50.631017 12663 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0507 06:32:50.632732 12663 ts_tablet_manager.cc:356] Loaded tablet metadata (0 total tablets, 0 live tablets) I0507 06:32:50.632805 12663 ts_tablet_manager.cc:376] Registered 0 tablets I0507 06:32:50.667273 12663 rpc_server.cc:205] RPC server started. Bound to: 127.0.57.193:37613 I0507 06:32:50.668669 12663 webserver.cc:308] Webserver started at http://127.0.57.193:40869/ using document root and password file I0507 06:32:50.669472 12663 server_base.cc:608] Dumped server information to /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_0.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data/info.pb I0507 06:32:50.674064 231 external_mini_cluster.cc:995] Started /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver as pid 12663 I0507 06:32:50.674741 231 external_mini_cluster.cc:933] Running /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_0.1557210500371863-231/raft_consensus-itest-cluster/ts-1/wal --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_0.1557210500371863-231/raft_consensus-itest-cluster/ts-1/data --block_manager=log --rpc_bind_addresses=127.0.57.194:0 --local_ip_for_outbound_sockets=127.0.57.194 --webserver_interface=127.0.57.194 --webserver_port=0 --tserver_master_addrs=127.0.57.254:36417 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_0.1557210500371863-231/raft_consensus-itest-cluster/ts-1/logs --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_0.1557210500371863-231/raft_consensus-itest-cluster/ts-1/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=true --follower_unavailable_considered_failed_sec=5 --log_compression_codec=none --log_cache_size_limit_mb=1 --log_segment_size_mb=1 --log_async_preallocate_segments=false --log_min_segments_to_retain=1 --log_max_segments_to_retain=3 --maintenance_manager_polling_interval_ms=100 --log_target_replay_size_mb=1 --max_cell_size_bytes=1000000 with env {} I0507 06:32:50.686656 12792 heartbeater.cc:345] Connected to a master server at 127.0.57.254:36417 I0507 06:32:50.686882 12792 heartbeater.cc:426] Registering TS with master... I0507 06:32:50.687438 12792 heartbeater.cc:475] Master 127.0.57.254:36417 requested a full tablet report, sending... I0507 06:32:50.689296 12629 ts_manager.cc:159] Registered new tserver with Master: e167493e37494c2896e3b5fe35a5a30d (127.0.57.193:37613) I0507 06:32:50.691035 12629 master_service.cc:273] Signed X509 certificate for tserver {username='slave'} at 127.0.57.193:59167 WARNING: Logging before InitGoogleLogging() is written to STDERR W0507 06:32:50.848104 12796 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0507 06:32:50.848369 12796 flags.cc:404] Enabled unsafe flag: --max_cell_size_bytes=1000000 W0507 06:32:50.848412 12796 flags.cc:404] Enabled unsafe flag: --never_fsync=true W0507 06:32:50.850589 12796 flags.cc:404] Enabled experimental flag: --log_compression_codec=none W0507 06:32:50.850656 12796 flags.cc:404] Enabled experimental flag: --log_max_segments_to_retain=3 W0507 06:32:50.850693 12796 flags.cc:404] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true W0507 06:32:50.850769 12796 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W0507 06:32:50.850847 12796 flags.cc:404] Enabled experimental flag: --log_target_replay_size_mb=1 W0507 06:32:50.850915 12796 flags.cc:404] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.57.194 I0507 06:32:50.855324 12796 tablet_server_main.cc:78] Tablet server non-default flags: --follower_unavailable_considered_failed_sec=5 --log_compression_codec=none --log_max_segments_to_retain=3 --log_cache_size_limit_mb=1 --log_async_preallocate_segments=false --log_segment_size_mb=1 --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_0.1557210500371863-231/raft_consensus-itest-cluster/ts-1/data --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_0.1557210500371863-231/raft_consensus-itest-cluster/ts-1/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.0.57.194:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_0.1557210500371863-231/raft_consensus-itest-cluster/ts-1/data/info.pb --webserver_interface=127.0.57.194 --webserver_port=0 --max_cell_size_bytes=1000000 --tserver_master_addrs=127.0.57.254:36417 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --log_target_replay_size_mb=1 --maintenance_manager_polling_interval_ms=100 --enable_minidumps=false --local_ip_for_outbound_sockets=127.0.57.194 --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_0.1557210500371863-231/raft_consensus-itest-cluster/ts-1/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.10.0-cdh6.x-SNAPSHOT revision 7d0ee26697926e87af9ef395008799577f6ab43f build type FASTDEBUG built by jenkins at 06 May 2019 22:44:21 PST on quasar-knmpfx-1.vpc.cloudera.com build id 322017 ASAN enabled I0507 06:32:50.863833 12796 system_ntp.cc:202] NTP initialized. Skew: 500ppm Current error: 159516us I0507 06:32:50.864647 12796 fs_manager.cc:263] Metadata directory not provided I0507 06:32:50.864717 12796 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0507 06:32:50.865025 12796 server_base.cc:433] Could not load existing FS layout: Not found: could not find a healthy instance file I0507 06:32:50.865098 12796 server_base.cc:434] Attempting to create new FS layout instead I0507 06:32:50.867694 12796 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_0.1557210500371863-231/raft_consensus-itest-cluster/ts-1/data/instance: uuid: "598969e0661641f7839d6402bd26c612" format_stamp: "Formatted at 2019-05-07 06:32:50 on dist-test-slave-dist-test-slave-rqp6" I0507 06:32:50.868278 12796 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_0.1557210500371863-231/raft_consensus-itest-cluster/ts-1/wal/instance: uuid: "598969e0661641f7839d6402bd26c612" format_stamp: "Formatted at 2019-05-07 06:32:50 on dist-test-slave-dist-test-slave-rqp6" I0507 06:32:50.875646 12796 fs_manager.cc:503] Time spent creating directory manager: real 0.007s user 0.009s sys 0.000s I0507 06:32:50.876839 12796 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0507 06:32:50.877496 12796 file_cache.cc:466] Constructed file cache lbm with capacity 419430 I0507 06:32:50.880705 12796 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s I0507 06:32:50.880820 12796 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_0.1557210500371863-231/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_0.1557210500371863-231/raft_consensus-itest-cluster/ts-1/wal uuid: "598969e0661641f7839d6402bd26c612" format_stamp: "Formatted at 2019-05-07 06:32:50 on dist-test-slave-dist-test-slave-rqp6" I0507 06:32:50.881002 12796 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_0.1557210500371863-231/raft_consensus-itest-cluster/ts-1/wal metadata directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_0.1557210500371863-231/raft_consensus-itest-cluster/ts-1/wal 1 data directories: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_0.1557210500371863-231/raft_consensus-itest-cluster/ts-1/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0507 06:32:50.959995 12796 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0507 06:32:50.961941 12796 ts_tablet_manager.cc:356] Loaded tablet metadata (0 total tablets, 0 live tablets) I0507 06:32:50.962026 12796 ts_tablet_manager.cc:376] Registered 0 tablets I0507 06:32:50.989418 12796 rpc_server.cc:205] RPC server started. Bound to: 127.0.57.194:33509 I0507 06:32:50.990589 12796 webserver.cc:308] Webserver started at http://127.0.57.194:36625/ using document root and password file I0507 06:32:50.991295 12796 server_base.cc:608] Dumped server information to /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_0.1557210500371863-231/raft_consensus-itest-cluster/ts-1/data/info.pb I0507 06:32:50.997390 231 external_mini_cluster.cc:995] Started /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver as pid 12796 I0507 06:32:50.998116 231 external_mini_cluster.cc:933] Running /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_0.1557210500371863-231/raft_consensus-itest-cluster/ts-2/wal --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_0.1557210500371863-231/raft_consensus-itest-cluster/ts-2/data --block_manager=log --rpc_bind_addresses=127.0.57.195:0 --local_ip_for_outbound_sockets=127.0.57.195 --webserver_interface=127.0.57.195 --webserver_port=0 --tserver_master_addrs=127.0.57.254:36417 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_0.1557210500371863-231/raft_consensus-itest-cluster/ts-2/logs --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_0.1557210500371863-231/raft_consensus-itest-cluster/ts-2/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=true --follower_unavailable_considered_failed_sec=5 --log_compression_codec=none --log_cache_size_limit_mb=1 --log_segment_size_mb=1 --log_async_preallocate_segments=false --log_min_segments_to_retain=1 --log_max_segments_to_retain=3 --maintenance_manager_polling_interval_ms=100 --log_target_replay_size_mb=1 --max_cell_size_bytes=1000000 with env {} I0507 06:32:51.007176 12915 heartbeater.cc:345] Connected to a master server at 127.0.57.254:36417 I0507 06:32:51.007436 12915 heartbeater.cc:426] Registering TS with master... I0507 06:32:51.008018 12915 heartbeater.cc:475] Master 127.0.57.254:36417 requested a full tablet report, sending... I0507 06:32:51.009306 12629 ts_manager.cc:159] Registered new tserver with Master: 598969e0661641f7839d6402bd26c612 (127.0.57.194:33509) I0507 06:32:51.010308 12629 master_service.cc:273] Signed X509 certificate for tserver {username='slave'} at 127.0.57.194:51307 WARNING: Logging before InitGoogleLogging() is written to STDERR W0507 06:32:51.174803 12918 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0507 06:32:51.175016 12918 flags.cc:404] Enabled unsafe flag: --max_cell_size_bytes=1000000 W0507 06:32:51.175060 12918 flags.cc:404] Enabled unsafe flag: --never_fsync=true W0507 06:32:51.177219 12918 flags.cc:404] Enabled experimental flag: --log_compression_codec=none W0507 06:32:51.177279 12918 flags.cc:404] Enabled experimental flag: --log_max_segments_to_retain=3 W0507 06:32:51.177425 12918 flags.cc:404] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true W0507 06:32:51.177480 12918 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W0507 06:32:51.177561 12918 flags.cc:404] Enabled experimental flag: --log_target_replay_size_mb=1 W0507 06:32:51.177609 12918 flags.cc:404] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.57.195 I0507 06:32:51.181833 12918 tablet_server_main.cc:78] Tablet server non-default flags: --follower_unavailable_considered_failed_sec=5 --log_compression_codec=none --log_max_segments_to_retain=3 --log_cache_size_limit_mb=1 --log_async_preallocate_segments=false --log_segment_size_mb=1 --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_0.1557210500371863-231/raft_consensus-itest-cluster/ts-2/data --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_0.1557210500371863-231/raft_consensus-itest-cluster/ts-2/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.0.57.195:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_0.1557210500371863-231/raft_consensus-itest-cluster/ts-2/data/info.pb --webserver_interface=127.0.57.195 --webserver_port=0 --max_cell_size_bytes=1000000 --tserver_master_addrs=127.0.57.254:36417 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --log_target_replay_size_mb=1 --maintenance_manager_polling_interval_ms=100 --enable_minidumps=false --local_ip_for_outbound_sockets=127.0.57.195 --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_0.1557210500371863-231/raft_consensus-itest-cluster/ts-2/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.10.0-cdh6.x-SNAPSHOT revision 7d0ee26697926e87af9ef395008799577f6ab43f build type FASTDEBUG built by jenkins at 06 May 2019 22:44:21 PST on quasar-knmpfx-1.vpc.cloudera.com build id 322017 ASAN enabled I0507 06:32:51.190127 12918 system_ntp.cc:202] NTP initialized. Skew: 500ppm Current error: 160016us I0507 06:32:51.190819 12918 fs_manager.cc:263] Metadata directory not provided I0507 06:32:51.190883 12918 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0507 06:32:51.191141 12918 server_base.cc:433] Could not load existing FS layout: Not found: could not find a healthy instance file I0507 06:32:51.191207 12918 server_base.cc:434] Attempting to create new FS layout instead I0507 06:32:51.193856 12918 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_0.1557210500371863-231/raft_consensus-itest-cluster/ts-2/data/instance: uuid: "de47b12769c24f8489d00f790f475b8d" format_stamp: "Formatted at 2019-05-07 06:32:51 on dist-test-slave-dist-test-slave-rqp6" I0507 06:32:51.194540 12918 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_0.1557210500371863-231/raft_consensus-itest-cluster/ts-2/wal/instance: uuid: "de47b12769c24f8489d00f790f475b8d" format_stamp: "Formatted at 2019-05-07 06:32:51 on dist-test-slave-dist-test-slave-rqp6" I0507 06:32:51.201704 12918 fs_manager.cc:503] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s I0507 06:32:51.202915 12918 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0507 06:32:51.203529 12918 file_cache.cc:466] Constructed file cache lbm with capacity 419430 I0507 06:32:51.207145 12918 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.000s sys 0.000s I0507 06:32:51.207238 12918 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_0.1557210500371863-231/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_0.1557210500371863-231/raft_consensus-itest-cluster/ts-2/wal uuid: "de47b12769c24f8489d00f790f475b8d" format_stamp: "Formatted at 2019-05-07 06:32:51 on dist-test-slave-dist-test-slave-rqp6" I0507 06:32:51.207408 12918 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_0.1557210500371863-231/raft_consensus-itest-cluster/ts-2/wal metadata directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_0.1557210500371863-231/raft_consensus-itest-cluster/ts-2/wal 1 data directories: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_0.1557210500371863-231/raft_consensus-itest-cluster/ts-2/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0507 06:32:51.232213 12918 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0507 06:32:51.233960 12918 ts_tablet_manager.cc:356] Loaded tablet metadata (0 total tablets, 0 live tablets) I0507 06:32:51.234043 12918 ts_tablet_manager.cc:376] Registered 0 tablets I0507 06:32:51.262853 12918 rpc_server.cc:205] RPC server started. Bound to: 127.0.57.195:36909 I0507 06:32:51.264142 12918 webserver.cc:308] Webserver started at http://127.0.57.195:45723/ using document root and password file I0507 06:32:51.264950 12918 server_base.cc:608] Dumped server information to /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_0.1557210500371863-231/raft_consensus-itest-cluster/ts-2/data/info.pb I0507 06:32:51.270895 231 external_mini_cluster.cc:995] Started /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver as pid 12918 I0507 06:32:51.278853 13037 heartbeater.cc:345] Connected to a master server at 127.0.57.254:36417 I0507 06:32:51.279093 13037 heartbeater.cc:426] Registering TS with master... I0507 06:32:51.279654 13037 heartbeater.cc:475] Master 127.0.57.254:36417 requested a full tablet report, sending... I0507 06:32:51.281134 12628 ts_manager.cc:159] Registered new tserver with Master: de47b12769c24f8489d00f790f475b8d (127.0.57.195:36909) I0507 06:32:51.282271 231 external_mini_cluster.cc:623] 3 TS(s) registered with all masters I0507 06:32:51.287832 12628 master_service.cc:273] Signed X509 certificate for tserver {username='slave'} at 127.0.57.195:40441 I0507 06:32:51.295570 12628 catalog_manager.cc:1415] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49598: name: "TestTable" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 3 split_rows_range_bounds { } partition_schema { range_schema { columns { name: "key" } } } W0507 06:32:51.300315 12628 catalog_manager.cc:1574] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestTable in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers. I0507 06:32:51.326768 12852 tablet_service.cc:1016] Processing CreateTablet for tablet 78673d8ea75d4380b30b00865393ae40 (table=TestTable [id=1c9be74812b944c89075f47edc5e7f8b]), partition=RANGE (key) PARTITION UNBOUNDED I0507 06:32:51.327021 12729 tablet_service.cc:1016] Processing CreateTablet for tablet 78673d8ea75d4380b30b00865393ae40 (table=TestTable [id=1c9be74812b944c89075f47edc5e7f8b]), partition=RANGE (key) PARTITION UNBOUNDED I0507 06:32:51.327060 12974 tablet_service.cc:1016] Processing CreateTablet for tablet 78673d8ea75d4380b30b00865393ae40 (table=TestTable [id=1c9be74812b944c89075f47edc5e7f8b]), partition=RANGE (key) PARTITION UNBOUNDED I0507 06:32:51.327821 12852 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet 78673d8ea75d4380b30b00865393ae40. 1 dirs total, 0 dirs full, 0 dirs failed I0507 06:32:51.327857 12974 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet 78673d8ea75d4380b30b00865393ae40. 1 dirs total, 0 dirs full, 0 dirs failed I0507 06:32:51.327862 12729 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet 78673d8ea75d4380b30b00865393ae40. 1 dirs total, 0 dirs full, 0 dirs failed I0507 06:32:51.336773 13057 tablet_bootstrap.cc:439] T 78673d8ea75d4380b30b00865393ae40 P de47b12769c24f8489d00f790f475b8d: Bootstrap starting. I0507 06:32:51.338950 13057 tablet_bootstrap.cc:590] T 78673d8ea75d4380b30b00865393ae40 P de47b12769c24f8489d00f790f475b8d: No blocks or log segments found. Creating new log. I0507 06:32:51.339778 13057 log.cc:525] T 78673d8ea75d4380b30b00865393ae40 P de47b12769c24f8489d00f790f475b8d: Log is configured to *not* fsync() on all Append() calls I0507 06:32:51.341871 13058 tablet_bootstrap.cc:439] T 78673d8ea75d4380b30b00865393ae40 P 598969e0661641f7839d6402bd26c612: Bootstrap starting. I0507 06:32:51.344035 13058 tablet_bootstrap.cc:590] T 78673d8ea75d4380b30b00865393ae40 P 598969e0661641f7839d6402bd26c612: No blocks or log segments found. Creating new log. I0507 06:32:51.345183 13056 tablet_bootstrap.cc:439] T 78673d8ea75d4380b30b00865393ae40 P e167493e37494c2896e3b5fe35a5a30d: Bootstrap starting. I0507 06:32:51.347064 13056 tablet_bootstrap.cc:590] T 78673d8ea75d4380b30b00865393ae40 P e167493e37494c2896e3b5fe35a5a30d: No blocks or log segments found. Creating new log. I0507 06:32:51.347858 13056 log.cc:525] T 78673d8ea75d4380b30b00865393ae40 P e167493e37494c2896e3b5fe35a5a30d: Log is configured to *not* fsync() on all Append() calls I0507 06:32:51.348438 13058 log.cc:525] T 78673d8ea75d4380b30b00865393ae40 P 598969e0661641f7839d6402bd26c612: Log is configured to *not* fsync() on all Append() calls I0507 06:32:51.349997 13057 tablet_bootstrap.cc:439] T 78673d8ea75d4380b30b00865393ae40 P de47b12769c24f8489d00f790f475b8d: No bootstrap required, opened a new log I0507 06:32:51.350260 13058 tablet_bootstrap.cc:439] T 78673d8ea75d4380b30b00865393ae40 P 598969e0661641f7839d6402bd26c612: No bootstrap required, opened a new log I0507 06:32:51.350260 13056 tablet_bootstrap.cc:439] T 78673d8ea75d4380b30b00865393ae40 P e167493e37494c2896e3b5fe35a5a30d: No bootstrap required, opened a new log I0507 06:32:51.350317 13057 ts_tablet_manager.cc:1053] T 78673d8ea75d4380b30b00865393ae40 P de47b12769c24f8489d00f790f475b8d: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.000s I0507 06:32:51.350507 13058 ts_tablet_manager.cc:1053] T 78673d8ea75d4380b30b00865393ae40 P 598969e0661641f7839d6402bd26c612: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.000s I0507 06:32:51.350515 13056 ts_tablet_manager.cc:1053] T 78673d8ea75d4380b30b00865393ae40 P e167493e37494c2896e3b5fe35a5a30d: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s I0507 06:32:51.355836 13058 raft_consensus.cc:340] T 78673d8ea75d4380b30b00865393ae40 P 598969e0661641f7839d6402bd26c612 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de47b12769c24f8489d00f790f475b8d" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 36909 } } peers { permanent_uuid: "e167493e37494c2896e3b5fe35a5a30d" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 37613 } } peers { permanent_uuid: "598969e0661641f7839d6402bd26c612" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 33509 } } I0507 06:32:51.355836 13056 raft_consensus.cc:340] T 78673d8ea75d4380b30b00865393ae40 P e167493e37494c2896e3b5fe35a5a30d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de47b12769c24f8489d00f790f475b8d" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 36909 } } peers { permanent_uuid: "e167493e37494c2896e3b5fe35a5a30d" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 37613 } } peers { permanent_uuid: "598969e0661641f7839d6402bd26c612" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 33509 } } I0507 06:32:51.356146 13056 raft_consensus.cc:366] T 78673d8ea75d4380b30b00865393ae40 P e167493e37494c2896e3b5fe35a5a30d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0507 06:32:51.356148 13058 raft_consensus.cc:366] T 78673d8ea75d4380b30b00865393ae40 P 598969e0661641f7839d6402bd26c612 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0507 06:32:51.356187 13056 raft_consensus.cc:705] T 78673d8ea75d4380b30b00865393ae40 P e167493e37494c2896e3b5fe35a5a30d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e167493e37494c2896e3b5fe35a5a30d, State: Initialized, Role: FOLLOWER I0507 06:32:51.356189 13058 raft_consensus.cc:705] T 78673d8ea75d4380b30b00865393ae40 P 598969e0661641f7839d6402bd26c612 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 598969e0661641f7839d6402bd26c612, State: Initialized, Role: FOLLOWER I0507 06:32:51.356189 13057 raft_consensus.cc:340] T 78673d8ea75d4380b30b00865393ae40 P de47b12769c24f8489d00f790f475b8d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de47b12769c24f8489d00f790f475b8d" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 36909 } } peers { permanent_uuid: "e167493e37494c2896e3b5fe35a5a30d" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 37613 } } peers { permanent_uuid: "598969e0661641f7839d6402bd26c612" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 33509 } } I0507 06:32:51.356559 13057 raft_consensus.cc:366] T 78673d8ea75d4380b30b00865393ae40 P de47b12769c24f8489d00f790f475b8d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0507 06:32:51.356608 13057 raft_consensus.cc:705] T 78673d8ea75d4380b30b00865393ae40 P de47b12769c24f8489d00f790f475b8d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de47b12769c24f8489d00f790f475b8d, State: Initialized, Role: FOLLOWER I0507 06:32:51.356681 13058 consensus_queue.cc:229] T 78673d8ea75d4380b30b00865393ae40 P 598969e0661641f7839d6402bd26c612 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de47b12769c24f8489d00f790f475b8d" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 36909 } } peers { permanent_uuid: "e167493e37494c2896e3b5fe35a5a30d" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 37613 } } peers { permanent_uuid: "598969e0661641f7839d6402bd26c612" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 33509 } } I0507 06:32:51.356686 13056 consensus_queue.cc:229] T 78673d8ea75d4380b30b00865393ae40 P e167493e37494c2896e3b5fe35a5a30d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de47b12769c24f8489d00f790f475b8d" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 36909 } } peers { permanent_uuid: "e167493e37494c2896e3b5fe35a5a30d" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 37613 } } peers { permanent_uuid: "598969e0661641f7839d6402bd26c612" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 33509 } } I0507 06:32:51.357023 13057 consensus_queue.cc:229] T 78673d8ea75d4380b30b00865393ae40 P de47b12769c24f8489d00f790f475b8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de47b12769c24f8489d00f790f475b8d" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 36909 } } peers { permanent_uuid: "e167493e37494c2896e3b5fe35a5a30d" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 37613 } } peers { permanent_uuid: "598969e0661641f7839d6402bd26c612" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 33509 } } I0507 06:32:51.357475 12915 heartbeater.cc:467] Master 127.0.57.254:36417 was elected leader, sending a full tablet report... I0507 06:32:51.357625 12792 heartbeater.cc:467] Master 127.0.57.254:36417 was elected leader, sending a full tablet report... I0507 06:32:51.358055 13037 heartbeater.cc:467] Master 127.0.57.254:36417 was elected leader, sending a full tablet report... I0507 06:32:51.358242 13058 ts_tablet_manager.cc:1081] T 78673d8ea75d4380b30b00865393ae40 P 598969e0661641f7839d6402bd26c612: Time spent starting tablet: real 0.008s user 0.004s sys 0.003s I0507 06:32:51.358897 13057 ts_tablet_manager.cc:1081] T 78673d8ea75d4380b30b00865393ae40 P de47b12769c24f8489d00f790f475b8d: Time spent starting tablet: real 0.009s user 0.005s sys 0.004s I0507 06:32:51.359653 13056 ts_tablet_manager.cc:1081] T 78673d8ea75d4380b30b00865393ae40 P e167493e37494c2896e3b5fe35a5a30d: Time spent starting tablet: real 0.009s user 0.008s sys 0.000s W0507 06:32:51.366881 13038 tablet.cc:1786] T 78673d8ea75d4380b30b00865393ae40 P de47b12769c24f8489d00f790f475b8d: Can't schedule compaction. Clean time has not been advanced past its initial value. W0507 06:32:51.372419 12793 tablet.cc:1786] T 78673d8ea75d4380b30b00865393ae40 P e167493e37494c2896e3b5fe35a5a30d: Can't schedule compaction. Clean time has not been advanced past its initial value. W0507 06:32:51.393851 12916 tablet.cc:1786] T 78673d8ea75d4380b30b00865393ae40 P 598969e0661641f7839d6402bd26c612: Can't schedule compaction. Clean time has not been advanced past its initial value. I0507 06:32:51.571758 13063 raft_consensus.cc:472] T 78673d8ea75d4380b30b00865393ae40 P e167493e37494c2896e3b5fe35a5a30d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0507 06:32:51.572006 13063 raft_consensus.cc:494] T 78673d8ea75d4380b30b00865393ae40 P e167493e37494c2896e3b5fe35a5a30d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de47b12769c24f8489d00f790f475b8d" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 36909 } } peers { permanent_uuid: "e167493e37494c2896e3b5fe35a5a30d" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 37613 } } peers { permanent_uuid: "598969e0661641f7839d6402bd26c612" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 33509 } } I0507 06:32:51.573348 13063 leader_election.cc:296] T 78673d8ea75d4380b30b00865393ae40 P e167493e37494c2896e3b5fe35a5a30d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers de47b12769c24f8489d00f790f475b8d (127.0.57.195:36909), 598969e0661641f7839d6402bd26c612 (127.0.57.194:33509) I0507 06:32:51.582881 12872 tablet_service.cc:1286] Received RequestConsensusVote() RPC: tablet_id: "78673d8ea75d4380b30b00865393ae40" candidate_uuid: "e167493e37494c2896e3b5fe35a5a30d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "598969e0661641f7839d6402bd26c612" is_pre_election: true I0507 06:32:51.582890 12994 tablet_service.cc:1286] Received RequestConsensusVote() RPC: tablet_id: "78673d8ea75d4380b30b00865393ae40" candidate_uuid: "e167493e37494c2896e3b5fe35a5a30d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de47b12769c24f8489d00f790f475b8d" is_pre_election: true I0507 06:32:51.583257 12872 raft_consensus.cc:2346] T 78673d8ea75d4380b30b00865393ae40 P 598969e0661641f7839d6402bd26c612 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e167493e37494c2896e3b5fe35a5a30d in term 0. I0507 06:32:51.583258 12994 raft_consensus.cc:2346] T 78673d8ea75d4380b30b00865393ae40 P de47b12769c24f8489d00f790f475b8d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e167493e37494c2896e3b5fe35a5a30d in term 0. I0507 06:32:51.583963 12680 leader_election.cc:310] T 78673d8ea75d4380b30b00865393ae40 P e167493e37494c2896e3b5fe35a5a30d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 598969e0661641f7839d6402bd26c612, e167493e37494c2896e3b5fe35a5a30d; no voters: I0507 06:32:51.584554 13063 raft_consensus.cc:2640] T 78673d8ea75d4380b30b00865393ae40 P e167493e37494c2896e3b5fe35a5a30d [term 0 FOLLOWER]: Leader pre-election won for term 1 I0507 06:32:51.584666 13063 raft_consensus.cc:472] T 78673d8ea75d4380b30b00865393ae40 P e167493e37494c2896e3b5fe35a5a30d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0507 06:32:51.584725 13063 raft_consensus.cc:2886] T 78673d8ea75d4380b30b00865393ae40 P e167493e37494c2896e3b5fe35a5a30d [term 0 FOLLOWER]: Advancing to term 1 I0507 06:32:51.586539 13063 raft_consensus.cc:494] T 78673d8ea75d4380b30b00865393ae40 P e167493e37494c2896e3b5fe35a5a30d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de47b12769c24f8489d00f790f475b8d" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 36909 } } peers { permanent_uuid: "e167493e37494c2896e3b5fe35a5a30d" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 37613 } } peers { permanent_uuid: "598969e0661641f7839d6402bd26c612" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 33509 } } I0507 06:32:51.587123 13063 leader_election.cc:296] T 78673d8ea75d4380b30b00865393ae40 P e167493e37494c2896e3b5fe35a5a30d [CANDIDATE]: Term 1 election: Requested vote from peers de47b12769c24f8489d00f790f475b8d (127.0.57.195:36909), 598969e0661641f7839d6402bd26c612 (127.0.57.194:33509) I0507 06:32:51.587612 12994 tablet_service.cc:1286] Received RequestConsensusVote() RPC: tablet_id: "78673d8ea75d4380b30b00865393ae40" candidate_uuid: "e167493e37494c2896e3b5fe35a5a30d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de47b12769c24f8489d00f790f475b8d" I0507 06:32:51.587767 12994 raft_consensus.cc:2886] T 78673d8ea75d4380b30b00865393ae40 P de47b12769c24f8489d00f790f475b8d [term 0 FOLLOWER]: Advancing to term 1 I0507 06:32:51.587730 12872 tablet_service.cc:1286] Received RequestConsensusVote() RPC: tablet_id: "78673d8ea75d4380b30b00865393ae40" candidate_uuid: "e167493e37494c2896e3b5fe35a5a30d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "598969e0661641f7839d6402bd26c612" I0507 06:32:51.587885 12872 raft_consensus.cc:2886] T 78673d8ea75d4380b30b00865393ae40 P 598969e0661641f7839d6402bd26c612 [term 0 FOLLOWER]: Advancing to term 1 I0507 06:32:51.589716 12994 raft_consensus.cc:2346] T 78673d8ea75d4380b30b00865393ae40 P de47b12769c24f8489d00f790f475b8d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e167493e37494c2896e3b5fe35a5a30d in term 1. I0507 06:32:51.589802 12872 raft_consensus.cc:2346] T 78673d8ea75d4380b30b00865393ae40 P 598969e0661641f7839d6402bd26c612 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e167493e37494c2896e3b5fe35a5a30d in term 1. I0507 06:32:51.590147 12681 leader_election.cc:310] T 78673d8ea75d4380b30b00865393ae40 P e167493e37494c2896e3b5fe35a5a30d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: de47b12769c24f8489d00f790f475b8d, e167493e37494c2896e3b5fe35a5a30d; no voters: I0507 06:32:51.590494 13063 raft_consensus.cc:2640] T 78673d8ea75d4380b30b00865393ae40 P e167493e37494c2896e3b5fe35a5a30d [term 1 FOLLOWER]: Leader election won for term 1 I0507 06:32:51.590870 13063 raft_consensus.cc:667] T 78673d8ea75d4380b30b00865393ae40 P e167493e37494c2896e3b5fe35a5a30d [term 1 LEADER]: Becoming Leader. State: Replica: e167493e37494c2896e3b5fe35a5a30d, State: Running, Role: LEADER I0507 06:32:51.591255 13063 consensus_queue.cc:206] T 78673d8ea75d4380b30b00865393ae40 P e167493e37494c2896e3b5fe35a5a30d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de47b12769c24f8489d00f790f475b8d" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 36909 } } peers { permanent_uuid: "e167493e37494c2896e3b5fe35a5a30d" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 37613 } } peers { permanent_uuid: "598969e0661641f7839d6402bd26c612" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 33509 } } I0507 06:32:51.595465 12628 catalog_manager.cc:3988] T 78673d8ea75d4380b30b00865393ae40 P e167493e37494c2896e3b5fe35a5a30d reported cstate change: term changed from 0 to 1, leader changed from to e167493e37494c2896e3b5fe35a5a30d (127.0.57.193). New cstate: current_term: 1 leader_uuid: "e167493e37494c2896e3b5fe35a5a30d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de47b12769c24f8489d00f790f475b8d" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 36909 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e167493e37494c2896e3b5fe35a5a30d" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 37613 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "598969e0661641f7839d6402bd26c612" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 33509 } health_report { overall_health: UNKNOWN } } } I0507 06:32:51.617823 231 external_mini_cluster.cc:623] 3 TS(s) registered with all masters I0507 06:32:51.619292 231 ts_itest-base.cc:232] Waiting for 1 tablets on tserver e167493e37494c2896e3b5fe35a5a30d to finish bootstrapping I0507 06:32:51.626960 231 ts_itest-base.cc:232] Waiting for 1 tablets on tserver 598969e0661641f7839d6402bd26c612 to finish bootstrapping I0507 06:32:51.633385 231 ts_itest-base.cc:232] Waiting for 1 tablets on tserver de47b12769c24f8489d00f790f475b8d to finish bootstrapping I0507 06:32:51.654482 231 cluster_itest_util.cc:258] Not converged past 1 yet: 1.1 0.0 0.0 I0507 06:32:51.758842 231 cluster_itest_util.cc:258] Not converged past 1 yet: 1.1 0.0 0.0 I0507 06:32:51.963281 231 cluster_itest_util.cc:258] Not converged past 1 yet: 1.1 0.0 0.0 I0507 06:32:52.039844 12994 raft_consensus.cc:1181] T 78673d8ea75d4380b30b00865393ae40 P de47b12769c24f8489d00f790f475b8d [term 1 FOLLOWER]: Refusing update from remote peer e167493e37494c2896e3b5fe35a5a30d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch) I0507 06:32:52.040928 13063 consensus_queue.cc:984] T 78673d8ea75d4380b30b00865393ae40 P e167493e37494c2896e3b5fe35a5a30d [LEADER]: Connected to new peer: Peer: permanent_uuid: "de47b12769c24f8489d00f790f475b8d" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 36909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0507 06:32:52.054414 12872 raft_consensus.cc:1181] T 78673d8ea75d4380b30b00865393ae40 P 598969e0661641f7839d6402bd26c612 [term 1 FOLLOWER]: Refusing update from remote peer e167493e37494c2896e3b5fe35a5a30d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch) I0507 06:32:52.055358 13063 consensus_queue.cc:984] T 78673d8ea75d4380b30b00865393ae40 P e167493e37494c2896e3b5fe35a5a30d [LEADER]: Connected to new peer: Peer: permanent_uuid: "598969e0661641f7839d6402bd26c612" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 33509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0507 06:32:52.282439 231 test_util.cc:212] Using random seed: 1659640883 I0507 06:32:52.301296 231 test_workload.cc:306] TestWorkload: Skipping table creation because table TestTable already exists I0507 06:32:52.303303 231 raft_consensus-itest-base.cc:271] Waiting until we've written at least 4MB... I0507 06:32:52.377117 12793 maintenance_manager.cc:306] P e167493e37494c2896e3b5fe35a5a30d: Scheduling FlushMRSOp(78673d8ea75d4380b30b00865393ae40): 1840776 bytes log retention I0507 06:32:52.451556 12687 maintenance_manager.cc:501] P e167493e37494c2896e3b5fe35a5a30d: FlushMRSOp(78673d8ea75d4380b30b00865393ae40) complete. Timing: real 0.074s user 0.053s sys 0.004s Metrics: {"bytes_written":135803,"cfile_init":1,"compiler_manager_pool.queue_time_us":908,"data dirs.queue_time_us":6568,"data dirs.run_cpu_time_us":242,"data dirs.run_wall_time_us":794,"drs_written":1,"lbm_read_time_us":109,"lbm_reads_lt_1ms":4,"lbm_write_time_us":907,"lbm_writes_lt_1ms":23,"rows_written":14,"spinlock_wait_cycles":216320,"thread_start_us":3110,"threads_started":2} I0507 06:32:52.453366 12793 maintenance_manager.cc:306] P e167493e37494c2896e3b5fe35a5a30d: Scheduling LogGCOp(78673d8ea75d4380b30b00865393ae40): free 1840776 bytes of WAL I0507 06:32:52.453826 12687 log_reader.cc:377] T 78673d8ea75d4380b30b00865393ae40: removed 2 log segments from log reader I0507 06:32:52.453984 12687 log.cc:863] T 78673d8ea75d4380b30b00865393ae40 P e167493e37494c2896e3b5fe35a5a30d: Deleting log segment in path: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_0.1557210500371863-231/raft_consensus-itest-cluster/ts-0/wal/wals/78673d8ea75d4380b30b00865393ae40/wal-000000001 (ops 1-8) I0507 06:32:52.454177 12687 log.cc:863] T 78673d8ea75d4380b30b00865393ae40 P e167493e37494c2896e3b5fe35a5a30d: Deleting log segment in path: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_0.1557210500371863-231/raft_consensus-itest-cluster/ts-0/wal/wals/78673d8ea75d4380b30b00865393ae40/wal-000000002 (ops 9-15) I0507 06:32:52.454922 12687 maintenance_manager.cc:501] P e167493e37494c2896e3b5fe35a5a30d: LogGCOp(78673d8ea75d4380b30b00865393ae40) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {} I0507 06:32:52.455685 12793 maintenance_manager.cc:306] P e167493e37494c2896e3b5fe35a5a30d: Scheduling FlushMRSOp(78673d8ea75d4380b30b00865393ae40): 1840816 bytes log retention I0507 06:32:52.470113 231 raft_consensus-itest-base.cc:277] Waiting for log GC on e167493e37494c2896e3b5fe35a5a30d I0507 06:32:52.470419 231 raft_consensus-itest-base.cc:282] Log GC complete on e167493e37494c2896e3b5fe35a5a30d I0507 06:32:52.470993 13038 maintenance_manager.cc:306] P de47b12769c24f8489d00f790f475b8d: Scheduling FlushMRSOp(78673d8ea75d4380b30b00865393ae40): 4339002 bytes log retention I0507 06:32:52.517159 12687 maintenance_manager.cc:501] P e167493e37494c2896e3b5fe35a5a30d: FlushMRSOp(78673d8ea75d4380b30b00865393ae40) complete. Timing: real 0.061s user 0.048s sys 0.005s Metrics: {"bytes_written":135869,"cfile_init":1,"data dirs.queue_time_us":77,"data dirs.run_cpu_time_us":141,"data dirs.run_wall_time_us":575,"drs_written":1,"lbm_read_time_us":83,"lbm_reads_lt_1ms":4,"lbm_write_time_us":881,"lbm_writes_lt_1ms":23,"rows_written":19} I0507 06:32:52.518548 12793 maintenance_manager.cc:306] P e167493e37494c2896e3b5fe35a5a30d: Scheduling LogGCOp(78673d8ea75d4380b30b00865393ae40): free 920408 bytes of WAL I0507 06:32:52.518879 12687 log_reader.cc:377] T 78673d8ea75d4380b30b00865393ae40: removed 1 log segments from log reader I0507 06:32:52.518975 12687 log.cc:863] T 78673d8ea75d4380b30b00865393ae40 P e167493e37494c2896e3b5fe35a5a30d: Deleting log segment in path: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_0.1557210500371863-231/raft_consensus-itest-cluster/ts-0/wal/wals/78673d8ea75d4380b30b00865393ae40/wal-000000003 (ops 16-22) I0507 06:32:52.519409 12687 maintenance_manager.cc:501] P e167493e37494c2896e3b5fe35a5a30d: LogGCOp(78673d8ea75d4380b30b00865393ae40) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {} I0507 06:32:52.519767 12793 maintenance_manager.cc:306] P e167493e37494c2896e3b5fe35a5a30d: Scheduling UndoDeltaBlockGCOp(78673d8ea75d4380b30b00865393ae40): 645 bytes on disk I0507 06:32:52.520854 12687 maintenance_manager.cc:501] P e167493e37494c2896e3b5fe35a5a30d: UndoDeltaBlockGCOp(78673d8ea75d4380b30b00865393ae40) complete. Timing: real 0.001s user 0.001s sys 0.000s Metrics: {"cfile_init":2,"lbm_read_time_us":176,"lbm_reads_lt_1ms":8} I0507 06:32:52.521293 12793 maintenance_manager.cc:306] P e167493e37494c2896e3b5fe35a5a30d: Scheduling CompactRowSetsOp(78673d8ea75d4380b30b00865393ae40): perf score=0.890027 I0507 06:32:52.542075 12932 maintenance_manager.cc:501] P de47b12769c24f8489d00f790f475b8d: FlushMRSOp(78673d8ea75d4380b30b00865393ae40) complete. Timing: real 0.070s user 0.060s sys 0.008s Metrics: {"bytes_written":136001,"cfile_init":1,"compiler_manager_pool.queue_time_us":3921,"data dirs.queue_time_us":2736,"data dirs.run_cpu_time_us":197,"data dirs.run_wall_time_us":1118,"drs_written":1,"lbm_read_time_us":64,"lbm_reads_lt_1ms":4,"lbm_write_time_us":564,"lbm_writes_lt_1ms":23,"rows_written":37,"thread_start_us":5480,"threads_started":2} I0507 06:32:52.543303 13038 maintenance_manager.cc:306] P de47b12769c24f8489d00f790f475b8d: Scheduling LogGCOp(78673d8ea75d4380b30b00865393ae40): free 2498218 bytes of WAL I0507 06:32:52.543633 12932 log_reader.cc:377] T 78673d8ea75d4380b30b00865393ae40: removed 3 log segments from log reader I0507 06:32:52.543731 12932 log.cc:863] T 78673d8ea75d4380b30b00865393ae40 P de47b12769c24f8489d00f790f475b8d: Deleting log segment in path: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_0.1557210500371863-231/raft_consensus-itest-cluster/ts-2/wal/wals/78673d8ea75d4380b30b00865393ae40/wal-000000001 (ops 1-7) I0507 06:32:52.543843 12932 log.cc:863] T 78673d8ea75d4380b30b00865393ae40 P de47b12769c24f8489d00f790f475b8d: Deleting log segment in path: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_0.1557210500371863-231/raft_consensus-itest-cluster/ts-2/wal/wals/78673d8ea75d4380b30b00865393ae40/wal-000000002 (ops 8-13) I0507 06:32:52.543928 12932 log.cc:863] T 78673d8ea75d4380b30b00865393ae40 P de47b12769c24f8489d00f790f475b8d: Deleting log segment in path: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_0.1557210500371863-231/raft_consensus-itest-cluster/ts-2/wal/wals/78673d8ea75d4380b30b00865393ae40/wal-000000003 (ops 14-20) I0507 06:32:52.544548 12932 maintenance_manager.cc:501] P de47b12769c24f8489d00f790f475b8d: LogGCOp(78673d8ea75d4380b30b00865393ae40) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {} I0507 06:32:52.544807 13038 maintenance_manager.cc:306] P de47b12769c24f8489d00f790f475b8d: Scheduling UndoDeltaBlockGCOp(78673d8ea75d4380b30b00865393ae40): 551 bytes on disk I0507 06:32:52.545496 12932 maintenance_manager.cc:501] P de47b12769c24f8489d00f790f475b8d: UndoDeltaBlockGCOp(78673d8ea75d4380b30b00865393ae40) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":67,"lbm_reads_lt_1ms":4} I0507 06:32:52.646385 12687 maintenance_manager.cc:501] P e167493e37494c2896e3b5fe35a5a30d: CompactRowSetsOp(78673d8ea75d4380b30b00865393ae40) complete. Timing: real 0.125s user 0.100s sys 0.014s Metrics: {"bytes_written":136001,"cfile_cache_hit":10,"cfile_cache_hit_bytes":262392,"cfile_cache_miss":26,"cfile_cache_miss_bytes":264230,"cfile_init":5,"data dirs.queue_time_us":575,"data dirs.run_cpu_time_us":307,"data dirs.run_wall_time_us":783,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":1339,"lbm_reads_lt_1ms":46,"lbm_write_time_us":830,"lbm_writes_lt_1ms":23,"num_input_rowsets":2,"rows_written":33} I0507 06:32:52.647238 12793 maintenance_manager.cc:306] P e167493e37494c2896e3b5fe35a5a30d: Scheduling UndoDeltaBlockGCOp(78673d8ea75d4380b30b00865393ae40): 506 bytes on disk I0507 06:32:52.647778 12687 maintenance_manager.cc:501] P e167493e37494c2896e3b5fe35a5a30d: UndoDeltaBlockGCOp(78673d8ea75d4380b30b00865393ae40) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":53,"lbm_reads_lt_1ms":4} I0507 06:32:54.472759 231 raft_consensus-itest-base.cc:298] Servers converged with original term 1 I0507 06:32:54.472821 231 raft_consensus-itest-base.cc:303] Resuming 598969e0661641f7839d6402bd26c612 I0507 06:32:54.473922 13075 raft_consensus.cc:472] T 78673d8ea75d4380b30b00865393ae40 P 598969e0661641f7839d6402bd26c612 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e167493e37494c2896e3b5fe35a5a30d) I0507 06:32:54.475369 13075 raft_consensus.cc:494] T 78673d8ea75d4380b30b00865393ae40 P 598969e0661641f7839d6402bd26c612 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de47b12769c24f8489d00f790f475b8d" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 36909 } } peers { permanent_uuid: "e167493e37494c2896e3b5fe35a5a30d" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 37613 } } peers { permanent_uuid: "598969e0661641f7839d6402bd26c612" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 33509 } } I0507 06:32:54.476721 13075 leader_election.cc:296] T 78673d8ea75d4380b30b00865393ae40 P 598969e0661641f7839d6402bd26c612 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers de47b12769c24f8489d00f790f475b8d (127.0.57.195:36909), e167493e37494c2896e3b5fe35a5a30d (127.0.57.193:37613) I0507 06:32:54.486802 13103 consensus_queue.cc:689] T 78673d8ea75d4380b30b00865393ae40 P e167493e37494c2896e3b5fe35a5a30d [LEADER]: The logs necessary to catch up peer 598969e0661641f7839d6402bd26c612 have been garbage collected. The follower will never be able to catch up (Not found: Failed to read ops 3..32: Segment 1 which contained index 3 has been GCed) I0507 06:32:54.486902 13103 consensus_queue.cc:544] T 78673d8ea75d4380b30b00865393ae40 P e167493e37494c2896e3b5fe35a5a30d [LEADER]: The logs necessary to catch up peer 598969e0661641f7839d6402bd26c612 have been garbage collected. The replica will never be able to catch up I0507 06:32:54.488260 12994 tablet_service.cc:1286] Received RequestConsensusVote() RPC: tablet_id: "78673d8ea75d4380b30b00865393ae40" candidate_uuid: "598969e0661641f7839d6402bd26c612" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "de47b12769c24f8489d00f790f475b8d" is_pre_election: true I0507 06:32:54.488519 12994 raft_consensus.cc:2307] T 78673d8ea75d4380b30b00865393ae40 P de47b12769c24f8489d00f790f475b8d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 598969e0661641f7839d6402bd26c612 for term 2 because replica is either leader or believes a valid leader to be alive. I0507 06:32:54.489774 12749 tablet_service.cc:1286] Received RequestConsensusVote() RPC: tablet_id: "78673d8ea75d4380b30b00865393ae40" candidate_uuid: "598969e0661641f7839d6402bd26c612" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e167493e37494c2896e3b5fe35a5a30d" is_pre_election: true I0507 06:32:54.490025 12749 raft_consensus.cc:2307] T 78673d8ea75d4380b30b00865393ae40 P e167493e37494c2896e3b5fe35a5a30d [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate 598969e0661641f7839d6402bd26c612 for term 2 because replica is either leader or believes a valid leader to be alive. I0507 06:32:54.490029 12628 catalog_manager.cc:3739] Sending ChangeConfig:REMOVE_PEER on tablet 78673d8ea75d4380b30b00865393ae40 (attempt 1) I0507 06:32:54.490581 12805 leader_election.cc:310] T 78673d8ea75d4380b30b00865393ae40 P 598969e0661641f7839d6402bd26c612 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 598969e0661641f7839d6402bd26c612; no voters: de47b12769c24f8489d00f790f475b8d, e167493e37494c2896e3b5fe35a5a30d I0507 06:32:54.491037 13075 raft_consensus.cc:2588] T 78673d8ea75d4380b30b00865393ae40 P 598969e0661641f7839d6402bd26c612 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority I0507 06:32:54.491869 12749 consensus_queue.cc:206] T 78673d8ea75d4380b30b00865393ae40 P e167493e37494c2896e3b5fe35a5a30d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 38, Committed index: 38, Last appended: 1.38, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 39 OBSOLETE_local: false peers { permanent_uuid: "de47b12769c24f8489d00f790f475b8d" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 36909 } } peers { permanent_uuid: "e167493e37494c2896e3b5fe35a5a30d" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 37613 } } I0507 06:32:54.494130 12994 raft_consensus.cc:1181] T 78673d8ea75d4380b30b00865393ae40 P de47b12769c24f8489d00f790f475b8d [term 1 FOLLOWER]: Refusing update from remote peer e167493e37494c2896e3b5fe35a5a30d: Log matching property violated. Preceding OpId in replica: term: 1 index: 38. Preceding OpId from leader: term: 1 index: 39. (index mismatch) I0507 06:32:54.494690 13103 consensus_queue.cc:984] T 78673d8ea75d4380b30b00865393ae40 P e167493e37494c2896e3b5fe35a5a30d [LEADER]: Connected to new peer: Peer: permanent_uuid: "de47b12769c24f8489d00f790f475b8d" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 36909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 39, Last known committed idx: 38, Time since last communication: 0.000s I0507 06:32:54.497618 13110 raft_consensus.cc:2788] T 78673d8ea75d4380b30b00865393ae40 P e167493e37494c2896e3b5fe35a5a30d [term 1 LEADER]: Committing config change with OpId 1.39: config changed from index -1 to 39, VOTER 598969e0661641f7839d6402bd26c612 (127.0.57.194) evicted. New config: { opid_index: 39 OBSOLETE_local: false peers { permanent_uuid: "de47b12769c24f8489d00f790f475b8d" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 36909 } } peers { permanent_uuid: "e167493e37494c2896e3b5fe35a5a30d" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 37613 } } } I0507 06:32:54.498232 12994 raft_consensus.cc:2788] T 78673d8ea75d4380b30b00865393ae40 P de47b12769c24f8489d00f790f475b8d [term 1 FOLLOWER]: Committing config change with OpId 1.39: config changed from index -1 to 39, VOTER 598969e0661641f7839d6402bd26c612 (127.0.57.194) evicted. New config: { opid_index: 39 OBSOLETE_local: false peers { permanent_uuid: "de47b12769c24f8489d00f790f475b8d" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 36909 } } peers { permanent_uuid: "e167493e37494c2896e3b5fe35a5a30d" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 37613 } } } I0507 06:32:54.500890 12612 catalog_manager.cc:3531] ChangeConfig:REMOVE_PEER RPC for tablet 78673d8ea75d4380b30b00865393ae40 with cas_config_opid_index -1: ChangeConfig:REMOVE_PEER succeeded (attempt 1) I0507 06:32:54.501690 12628 catalog_manager.cc:3988] T 78673d8ea75d4380b30b00865393ae40 P e167493e37494c2896e3b5fe35a5a30d reported cstate change: config changed from index -1 to 39, VOTER 598969e0661641f7839d6402bd26c612 (127.0.57.194) evicted. New cstate: current_term: 1 leader_uuid: "e167493e37494c2896e3b5fe35a5a30d" committed_config { opid_index: 39 OBSOLETE_local: false peers { permanent_uuid: "de47b12769c24f8489d00f790f475b8d" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 36909 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e167493e37494c2896e3b5fe35a5a30d" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 37613 } health_report { overall_health: HEALTHY } } } I0507 06:32:54.505993 12628 catalog_manager.cc:3390] Sending DeleteTablet(TABLET_DATA_TOMBSTONED) for tablet 78673d8ea75d4380b30b00865393ae40 on 598969e0661641f7839d6402bd26c612 (127.0.57.194:33509) (TS 598969e0661641f7839d6402bd26c612 not found in new config with opid_index 39) I0507 06:32:54.506934 12852 tablet_service.cc:1057] Processing DeleteTablet for tablet 78673d8ea75d4380b30b00865393ae40 with delete_type TABLET_DATA_TOMBSTONED (TS 598969e0661641f7839d6402bd26c612 not found in new config with opid_index 39) from {username='slave'} at 127.0.0.1:57580 I0507 06:32:54.513022 13119 tablet_replica.cc:264] T 78673d8ea75d4380b30b00865393ae40 P 598969e0661641f7839d6402bd26c612: stopping tablet replica I0507 06:32:54.513329 13119 raft_consensus.cc:2143] T 78673d8ea75d4380b30b00865393ae40 P 598969e0661641f7839d6402bd26c612 [term 1 FOLLOWER]: Raft consensus shutting down. I0507 06:32:54.513581 13119 pending_rounds.cc:64] T 78673d8ea75d4380b30b00865393ae40 P 598969e0661641f7839d6402bd26c612: Trying to abort 1 pending transactions. I0507 06:32:54.513641 13119 pending_rounds.cc:70] T 78673d8ea75d4380b30b00865393ae40 P 598969e0661641f7839d6402bd26c612: Aborting transaction as it isn't in flight: id { term: 1 index: 2 } timestamp: 6378335323452145664 op_type: WRITE_OP write_request { tablet_id: "78673d8ea75d4380b30b00865393ae40" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true } } row_operations { rows: "\001\007\000\364\321\220\030a&\032\265\000\000\000\000\000\000\000\000\000\000\002\000\000\000\000\000" indirect_data: "0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 I0507 06:32:54.515722 13119 raft_consensus.cc:2170] T 78673d8ea75d4380b30b00865393ae40 P 598969e0661641f7839d6402bd26c612 [term 1 FOLLOWER]: Raft consensus is shut down! I0507 06:32:54.517408 13119 ts_tablet_manager.cc:1377] T 78673d8ea75d4380b30b00865393ae40 P 598969e0661641f7839d6402bd26c612: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED I0507 06:32:54.521028 13119 ts_tablet_manager.cc:1390] T 78673d8ea75d4380b30b00865393ae40 P 598969e0661641f7839d6402bd26c612: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2 I0507 06:32:54.521127 13119 log.cc:990] T 78673d8ea75d4380b30b00865393ae40 P 598969e0661641f7839d6402bd26c612: Deleting WAL directory at /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_0.1557210500371863-231/raft_consensus-itest-cluster/ts-1/wal/wals/78673d8ea75d4380b30b00865393ae40 I0507 06:32:54.521957 12610 catalog_manager.cc:3368] TS 598969e0661641f7839d6402bd26c612 (127.0.57.194:33509): tablet 78673d8ea75d4380b30b00865393ae40 (table TestTable [id=1c9be74812b944c89075f47edc5e7f8b]) successfully deleted I0507 06:32:54.524103 231 external_mini_cluster.cc:1172] Attempting to check leaks for /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-master pid 12603 I0507 06:32:54.524871 12659 generic_service.cc:171] Checking for leaks (request via RPC) I0507 06:32:54.553689 13038 maintenance_manager.cc:306] P de47b12769c24f8489d00f790f475b8d: Scheduling LogGCOp(78673d8ea75d4380b30b00865393ae40): free 1840784 bytes of WAL I0507 06:32:54.554128 12932 log_reader.cc:377] T 78673d8ea75d4380b30b00865393ae40: removed 2 log segments from log reader I0507 06:32:54.554193 12932 log.cc:863] T 78673d8ea75d4380b30b00865393ae40 P de47b12769c24f8489d00f790f475b8d: Deleting log segment in path: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_0.1557210500371863-231/raft_consensus-itest-cluster/ts-2/wal/wals/78673d8ea75d4380b30b00865393ae40/wal-000000004 (ops 21-27) I0507 06:32:54.554303 12932 log.cc:863] T 78673d8ea75d4380b30b00865393ae40 P de47b12769c24f8489d00f790f475b8d: Deleting log segment in path: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_0.1557210500371863-231/raft_consensus-itest-cluster/ts-2/wal/wals/78673d8ea75d4380b30b00865393ae40/wal-000000005 (ops 28-34) I0507 06:32:54.554777 12793 maintenance_manager.cc:306] P e167493e37494c2896e3b5fe35a5a30d: Scheduling LogGCOp(78673d8ea75d4380b30b00865393ae40): free 920408 bytes of WAL I0507 06:32:54.554774 12932 maintenance_manager.cc:501] P de47b12769c24f8489d00f790f475b8d: LogGCOp(78673d8ea75d4380b30b00865393ae40) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {} I0507 06:32:54.555171 12687 log_reader.cc:377] T 78673d8ea75d4380b30b00865393ae40: removed 1 log segments from log reader I0507 06:32:54.555249 12687 log.cc:863] T 78673d8ea75d4380b30b00865393ae40 P e167493e37494c2896e3b5fe35a5a30d: Deleting log segment in path: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_0.1557210500371863-231/raft_consensus-itest-cluster/ts-0/wal/wals/78673d8ea75d4380b30b00865393ae40/wal-000000004 (ops 23-29) I0507 06:32:54.555567 12687 maintenance_manager.cc:501] P e167493e37494c2896e3b5fe35a5a30d: LogGCOp(78673d8ea75d4380b30b00865393ae40) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {} I0507 06:32:54.602883 231 external_mini_cluster.cc:1131] Killing /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-master with pid 12603 I0507 06:32:54.619324 231 external_mini_cluster.cc:933] Running /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-master /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-master --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_0.1557210500371863-231/raft_consensus-itest-cluster/master-0/wal --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_0.1557210500371863-231/raft_consensus-itest-cluster/master-0/data --block_manager=log --webserver_interface=localhost --ipki_ca_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.0.57.254:36417 --webserver_interface=127.0.57.254 --webserver_port=32787 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_0.1557210500371863-231/raft_consensus-itest-cluster/master-0/logs --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_0.1557210500371863-231/raft_consensus-itest-cluster/master-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=true --master_add_server_when_underreplicated=false --rpc_reuseport=true --master_add_server_when_underreplicated=true with env {} WARNING: Logging before InitGoogleLogging() is written to STDERR W0507 06:32:54.835124 13121 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0507 06:32:54.835379 13121 flags.cc:404] Enabled unsafe flag: --never_fsync=true W0507 06:32:54.837918 13121 flags.cc:404] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true W0507 06:32:54.837994 13121 flags.cc:404] Enabled experimental flag: --ipki_ca_key_size=1024 W0507 06:32:54.838037 13121 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W0507 06:32:54.838058 13121 flags.cc:404] Enabled experimental flag: --tsk_num_rsa_bits=512 W0507 06:32:54.838109 13121 flags.cc:404] Enabled experimental flag: --rpc_reuseport=true I0507 06:32:54.842689 13121 master_main.cc:73] Master server non-default flags: --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_0.1557210500371863-231/raft_consensus-itest-cluster/master-0/data --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_0.1557210500371863-231/raft_consensus-itest-cluster/master-0/wal --ipki_ca_key_size=1024 --ipki_server_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.0.57.254:36417 --rpc_reuseport=true --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_0.1557210500371863-231/raft_consensus-itest-cluster/master-0/data/info.pb --webserver_interface=127.0.57.254 --webserver_port=32787 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_0.1557210500371863-231/raft_consensus-itest-cluster/master-0/logs --logbuflevel=-1 --logtostderr=true Master server version: kudu 1.10.0-cdh6.x-SNAPSHOT revision 7d0ee26697926e87af9ef395008799577f6ab43f build type FASTDEBUG built by jenkins at 06 May 2019 22:44:21 PST on quasar-knmpfx-1.vpc.cloudera.com build id 322017 ASAN enabled I0507 06:32:54.850080 13121 system_ntp.cc:202] NTP initialized. Skew: 500ppm Current error: 161516us I0507 06:32:54.851521 13121 fs_manager.cc:263] Metadata directory not provided I0507 06:32:54.851615 13121 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0507 06:32:54.858511 13121 fs_manager.cc:399] Time spent opening directory manager: real 0.005s user 0.003s sys 0.002s I0507 06:32:54.859213 13121 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0507 06:32:54.859822 13121 file_cache.cc:466] Constructed file cache lbm with capacity 419430 I0507 06:32:54.863101 13121 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s I0507 06:32:54.863209 13121 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_0.1557210500371863-231/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_0.1557210500371863-231/raft_consensus-itest-cluster/master-0/wal uuid: "a1bfe1b2ebf24d0ca70399022578708a" format_stamp: "Formatted at 2019-05-07 06:32:50 on dist-test-slave-dist-test-slave-rqp6" I0507 06:32:54.864307 13121 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_0.1557210500371863-231/raft_consensus-itest-cluster/master-0/wal metadata directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_0.1557210500371863-231/raft_consensus-itest-cluster/master-0/wal 1 data directories: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_0.1557210500371863-231/raft_consensus-itest-cluster/master-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0507 06:32:54.906733 13121 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0507 06:32:54.921735 13121 rpc_server.cc:205] RPC server started. Bound to: 127.0.57.254:36417 I0507 06:32:54.923210 13121 webserver.cc:308] Webserver started at http://127.0.57.254:32787/ using document root and password file I0507 06:32:54.924245 13121 server_base.cc:608] Dumped server information to /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_0.1557210500371863-231/raft_consensus-itest-cluster/master-0/data/info.pb I0507 06:32:54.931051 231 external_mini_cluster.cc:995] Started /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-master as pid 13121 I0507 06:32:54.934180 13180 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P a1bfe1b2ebf24d0ca70399022578708a: Bootstrap starting. I0507 06:32:54.953402 13180 log.cc:525] T 00000000000000000000000000000000 P a1bfe1b2ebf24d0ca70399022578708a: Log is configured to *not* fsync() on all Append() calls I0507 06:32:54.964301 13180 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P a1bfe1b2ebf24d0ca70399022578708a: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=4 ignored=0} mutations{seen=3 ignored=0} orphaned_commits=0. Pending: 0 replicates I0507 06:32:54.964874 13180 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P a1bfe1b2ebf24d0ca70399022578708a: Bootstrap complete. I0507 06:32:54.972471 13180 raft_consensus.cc:340] T 00000000000000000000000000000000 P a1bfe1b2ebf24d0ca70399022578708a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1bfe1b2ebf24d0ca70399022578708a" member_type: VOTER } I0507 06:32:54.973397 13180 raft_consensus.cc:705] T 00000000000000000000000000000000 P a1bfe1b2ebf24d0ca70399022578708a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1bfe1b2ebf24d0ca70399022578708a, State: Initialized, Role: FOLLOWER I0507 06:32:54.974040 13180 consensus_queue.cc:229] T 00000000000000000000000000000000 P a1bfe1b2ebf24d0ca70399022578708a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1bfe1b2ebf24d0ca70399022578708a" member_type: VOTER } I0507 06:32:54.974244 13180 raft_consensus.cc:380] T 00000000000000000000000000000000 P a1bfe1b2ebf24d0ca70399022578708a [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0507 06:32:54.974337 13180 raft_consensus.cc:472] T 00000000000000000000000000000000 P a1bfe1b2ebf24d0ca70399022578708a [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0507 06:32:54.974450 13180 raft_consensus.cc:2886] T 00000000000000000000000000000000 P a1bfe1b2ebf24d0ca70399022578708a [term 1 FOLLOWER]: Advancing to term 2 I0507 06:32:54.979041 13180 raft_consensus.cc:494] T 00000000000000000000000000000000 P a1bfe1b2ebf24d0ca70399022578708a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1bfe1b2ebf24d0ca70399022578708a" member_type: VOTER } I0507 06:32:54.979553 13180 leader_election.cc:310] T 00000000000000000000000000000000 P a1bfe1b2ebf24d0ca70399022578708a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a1bfe1b2ebf24d0ca70399022578708a; no voters: I0507 06:32:54.980363 13180 leader_election.cc:296] T 00000000000000000000000000000000 P a1bfe1b2ebf24d0ca70399022578708a [CANDIDATE]: Term 2 election: Requested vote from peers I0507 06:32:54.980602 13184 raft_consensus.cc:2640] T 00000000000000000000000000000000 P a1bfe1b2ebf24d0ca70399022578708a [term 2 FOLLOWER]: Leader election won for term 2 I0507 06:32:54.981675 13184 raft_consensus.cc:667] T 00000000000000000000000000000000 P a1bfe1b2ebf24d0ca70399022578708a [term 2 LEADER]: Becoming Leader. State: Replica: a1bfe1b2ebf24d0ca70399022578708a, State: Running, Role: LEADER I0507 06:32:54.982156 13184 consensus_queue.cc:206] T 00000000000000000000000000000000 P a1bfe1b2ebf24d0ca70399022578708a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1bfe1b2ebf24d0ca70399022578708a" member_type: VOTER } I0507 06:32:54.986038 13186 sys_catalog.cc:337] T 00000000000000000000000000000000 P a1bfe1b2ebf24d0ca70399022578708a [sys.catalog]: SysCatalogTable state changed. Reason: New leader a1bfe1b2ebf24d0ca70399022578708a. Latest consensus state: current_term: 2 leader_uuid: "a1bfe1b2ebf24d0ca70399022578708a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1bfe1b2ebf24d0ca70399022578708a" member_type: VOTER } } I0507 06:32:54.986460 13186 sys_catalog.cc:340] T 00000000000000000000000000000000 P a1bfe1b2ebf24d0ca70399022578708a [sys.catalog]: This master's current role is: LEADER I0507 06:32:54.986699 13185 sys_catalog.cc:337] T 00000000000000000000000000000000 P a1bfe1b2ebf24d0ca70399022578708a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "a1bfe1b2ebf24d0ca70399022578708a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1bfe1b2ebf24d0ca70399022578708a" member_type: VOTER } } I0507 06:32:54.987031 13180 sys_catalog.cc:337] T 00000000000000000000000000000000 P a1bfe1b2ebf24d0ca70399022578708a [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 2 leader_uuid: "a1bfe1b2ebf24d0ca70399022578708a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1bfe1b2ebf24d0ca70399022578708a" member_type: VOTER } } I0507 06:32:54.987190 13185 sys_catalog.cc:340] T 00000000000000000000000000000000 P a1bfe1b2ebf24d0ca70399022578708a [sys.catalog]: This master's current role is: LEADER I0507 06:32:54.987294 13180 sys_catalog.cc:340] T 00000000000000000000000000000000 P a1bfe1b2ebf24d0ca70399022578708a [sys.catalog]: This master's current role is: LEADER I0507 06:32:54.988639 13180 sys_catalog.cc:424] T 00000000000000000000000000000000 P a1bfe1b2ebf24d0ca70399022578708a [sys.catalog]: configured and running, proceeding with master startup. I0507 06:32:54.988875 13190 catalog_manager.cc:1055] Loading table and tablet metadata into memory... I0507 06:32:54.993567 13190 catalog_manager.cc:356] Loaded metadata for table TestTable [id=1c9be74812b944c89075f47edc5e7f8b] I0507 06:32:54.995234 13190 catalog_manager.cc:410] Loaded metadata for tablet 78673d8ea75d4380b30b00865393ae40 (table TestTable [id=1c9be74812b944c89075f47edc5e7f8b]) I0507 06:32:54.995393 13190 catalog_manager.cc:1066] Initializing Kudu internal certificate authority... I0507 06:32:54.998437 13190 catalog_manager.cc:1075] Loading token signing keys... I0507 06:32:54.999478 13190 catalog_manager.cc:4303] T 00000000000000000000000000000000 P a1bfe1b2ebf24d0ca70399022578708a: Loaded TSK: 0 I0507 06:32:55.483016 13147 master_service.cc:229] Got heartbeat from unknown tserver (permanent_uuid: "598969e0661641f7839d6402bd26c612" instance_seqno: 1557210770984106) as {username='slave'} at 127.0.57.194:39481; Asking this server to re-register. I0507 06:32:55.483871 12915 heartbeater.cc:426] Registering TS with master... I0507 06:32:55.484104 12915 heartbeater.cc:475] Master 127.0.57.254:36417 requested a full tablet report, sending... I0507 06:32:55.485442 13147 ts_manager.cc:159] Registered new tserver with Master: 598969e0661641f7839d6402bd26c612 (127.0.57.194:33509) I0507 06:32:55.512380 13147 master_service.cc:229] Got heartbeat from unknown tserver (permanent_uuid: "de47b12769c24f8489d00f790f475b8d" instance_seqno: 1557210771257229) as {username='slave'} at 127.0.57.195:60795; Asking this server to re-register. I0507 06:32:55.513036 13037 heartbeater.cc:426] Registering TS with master... I0507 06:32:55.513216 13037 heartbeater.cc:475] Master 127.0.57.254:36417 requested a full tablet report, sending... I0507 06:32:55.514248 13146 ts_manager.cc:159] Registered new tserver with Master: de47b12769c24f8489d00f790f475b8d (127.0.57.195:36909) I0507 06:32:55.514585 13147 master_service.cc:229] Got heartbeat from unknown tserver (permanent_uuid: "e167493e37494c2896e3b5fe35a5a30d" instance_seqno: 1557210770661819) as {username='slave'} at 127.0.57.193:43643; Asking this server to re-register. I0507 06:32:55.515199 12792 heartbeater.cc:426] Registering TS with master... I0507 06:32:55.515363 12792 heartbeater.cc:475] Master 127.0.57.254:36417 requested a full tablet report, sending... I0507 06:32:55.516283 13146 ts_manager.cc:159] Registered new tserver with Master: e167493e37494c2896e3b5fe35a5a30d (127.0.57.193:37613) I0507 06:32:55.517628 13146 catalog_manager.cc:3616] Sending ChangeConfig:ADD_PEER:NON_VOTER on tablet 78673d8ea75d4380b30b00865393ae40 (attempt 1) I0507 06:32:55.529976 12749 consensus_queue.cc:206] T 78673d8ea75d4380b30b00865393ae40 P e167493e37494c2896e3b5fe35a5a30d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 39, Committed index: 39, Last appended: 1.39, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 40 OBSOLETE_local: false peers { permanent_uuid: "de47b12769c24f8489d00f790f475b8d" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 36909 } } peers { permanent_uuid: "e167493e37494c2896e3b5fe35a5a30d" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 37613 } } peers { permanent_uuid: "598969e0661641f7839d6402bd26c612" member_type: NON_VOTER last_known_addr { host: "127.0.57.194" port: 33509 } attrs { promote: true } } I0507 06:32:55.536892 12994 raft_consensus.cc:1181] T 78673d8ea75d4380b30b00865393ae40 P de47b12769c24f8489d00f790f475b8d [term 1 FOLLOWER]: Refusing update from remote peer e167493e37494c2896e3b5fe35a5a30d: Log matching property violated. Preceding OpId in replica: term: 1 index: 39. Preceding OpId from leader: term: 1 index: 40. (index mismatch) W0507 06:32:55.536968 12680 consensus_peers.cc:457] T 78673d8ea75d4380b30b00865393ae40 P e167493e37494c2896e3b5fe35a5a30d -> Peer 598969e0661641f7839d6402bd26c612 (127.0.57.194:33509): Couldn't send request to peer 598969e0661641f7839d6402bd26c612. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry. I0507 06:32:55.537463 13201 consensus_queue.cc:984] T 78673d8ea75d4380b30b00865393ae40 P e167493e37494c2896e3b5fe35a5a30d [LEADER]: Connected to new peer: Peer: permanent_uuid: "de47b12769c24f8489d00f790f475b8d" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 36909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 40, Last known committed idx: 39, Time since last communication: 0.000s I0507 06:32:55.539981 13202 raft_consensus.cc:2788] T 78673d8ea75d4380b30b00865393ae40 P e167493e37494c2896e3b5fe35a5a30d [term 1 LEADER]: Committing config change with OpId 1.40: config changed from index 39 to 40, NON_VOTER 598969e0661641f7839d6402bd26c612 (127.0.57.194) added. New config: { opid_index: 40 OBSOLETE_local: false peers { permanent_uuid: "de47b12769c24f8489d00f790f475b8d" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 36909 } } peers { permanent_uuid: "e167493e37494c2896e3b5fe35a5a30d" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 37613 } } peers { permanent_uuid: "598969e0661641f7839d6402bd26c612" member_type: NON_VOTER last_known_addr { host: "127.0.57.194" port: 33509 } attrs { promote: true } } } I0507 06:32:55.540387 12994 raft_consensus.cc:2788] T 78673d8ea75d4380b30b00865393ae40 P de47b12769c24f8489d00f790f475b8d [term 1 FOLLOWER]: Committing config change with OpId 1.40: config changed from index 39 to 40, NON_VOTER 598969e0661641f7839d6402bd26c612 (127.0.57.194) added. New config: { opid_index: 40 OBSOLETE_local: false peers { permanent_uuid: "de47b12769c24f8489d00f790f475b8d" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 36909 } } peers { permanent_uuid: "e167493e37494c2896e3b5fe35a5a30d" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 37613 } } peers { permanent_uuid: "598969e0661641f7839d6402bd26c612" member_type: NON_VOTER last_known_addr { host: "127.0.57.194" port: 33509 } attrs { promote: true } } } I0507 06:32:55.542879 13130 catalog_manager.cc:3531] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 78673d8ea75d4380b30b00865393ae40 with cas_config_opid_index 39: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1) I0507 06:32:55.544190 13146 catalog_manager.cc:3988] T 78673d8ea75d4380b30b00865393ae40 P e167493e37494c2896e3b5fe35a5a30d reported cstate change: config changed from index 39 to 40, NON_VOTER 598969e0661641f7839d6402bd26c612 (127.0.57.194) added. New cstate: current_term: 1 leader_uuid: "e167493e37494c2896e3b5fe35a5a30d" committed_config { opid_index: 40 OBSOLETE_local: false peers { permanent_uuid: "de47b12769c24f8489d00f790f475b8d" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 36909 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e167493e37494c2896e3b5fe35a5a30d" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 37613 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "598969e0661641f7839d6402bd26c612" member_type: NON_VOTER last_known_addr { host: "127.0.57.194" port: 33509 } attrs { promote: true } health_report { overall_health: UNKNOWN } } } I0507 06:32:56.021056 13209 ts_tablet_manager.cc:676] T 78673d8ea75d4380b30b00865393ae40 P 598969e0661641f7839d6402bd26c612: Initiating tablet copy from peer e167493e37494c2896e3b5fe35a5a30d (127.0.57.193:37613) I0507 06:32:56.021456 13209 tablet_copy_client.cc:240] T 78673d8ea75d4380b30b00865393ae40 P 598969e0661641f7839d6402bd26c612: tablet copy: Beginning tablet copy session from remote peer at address 127.0.57.193:37613 I0507 06:32:56.022462 12769 tablet_copy_service.cc:135] P e167493e37494c2896e3b5fe35a5a30d: Received BeginTabletCopySession request for tablet 78673d8ea75d4380b30b00865393ae40 from peer 598969e0661641f7839d6402bd26c612 ({username='slave'} at 127.0.57.194:56019) I0507 06:32:56.022657 12769 tablet_copy_service.cc:156] P e167493e37494c2896e3b5fe35a5a30d: Beginning new tablet copy session on tablet 78673d8ea75d4380b30b00865393ae40 from peer 598969e0661641f7839d6402bd26c612 at {username='slave'} at 127.0.57.194:56019: session id = 598969e0661641f7839d6402bd26c612-78673d8ea75d4380b30b00865393ae40 I0507 06:32:56.025164 12769 tablet_copy_source_session.cc:202] T 78673d8ea75d4380b30b00865393ae40 P e167493e37494c2896e3b5fe35a5a30d: Tablet Copy: opened 5 blocks and 2 log segments I0507 06:32:56.027066 13209 ts_tablet_manager.cc:1377] T 78673d8ea75d4380b30b00865393ae40 P 598969e0661641f7839d6402bd26c612: Deleting tablet data with delete state TABLET_DATA_COPYING I0507 06:32:56.031103 13209 ts_tablet_manager.cc:1390] T 78673d8ea75d4380b30b00865393ae40 P 598969e0661641f7839d6402bd26c612: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.2 I0507 06:32:56.031333 13209 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet 78673d8ea75d4380b30b00865393ae40. 1 dirs total, 0 dirs full, 0 dirs failed I0507 06:32:56.033478 13209 tablet_copy_client.cc:575] T 78673d8ea75d4380b30b00865393ae40 P 598969e0661641f7839d6402bd26c612: tablet copy: Starting download of 5 data blocks... I0507 06:32:56.043051 13209 tablet_copy_client.cc:538] T 78673d8ea75d4380b30b00865393ae40 P 598969e0661641f7839d6402bd26c612: tablet copy: Starting download of 2 WAL segments... I0507 06:32:56.052105 13209 tablet_copy_client.cc:414] T 78673d8ea75d4380b30b00865393ae40 P 598969e0661641f7839d6402bd26c612: tablet copy: Tablet Copy complete. Replacing tablet superblock. I0507 06:32:56.055344 13209 tablet_bootstrap.cc:439] T 78673d8ea75d4380b30b00865393ae40 P 598969e0661641f7839d6402bd26c612: Bootstrap starting. I0507 06:32:56.072407 13209 tablet_bootstrap.cc:439] T 78673d8ea75d4380b30b00865393ae40 P 598969e0661641f7839d6402bd26c612: Bootstrap replayed 1/2 log segments. Stats: ops{read=7 overwritten=0 applied=5 ignored=5} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=2. Pending: 2 replicates I0507 06:32:56.080732 13209 tablet_bootstrap.cc:439] T 78673d8ea75d4380b30b00865393ae40 P 598969e0661641f7839d6402bd26c612: Bootstrap replayed 2/2 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=5} inserts{seen=4 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=2. Pending: 0 replicates I0507 06:32:56.081256 13209 tablet_bootstrap.cc:439] T 78673d8ea75d4380b30b00865393ae40 P 598969e0661641f7839d6402bd26c612: Bootstrap complete. I0507 06:32:56.081689 13209 ts_tablet_manager.cc:1053] T 78673d8ea75d4380b30b00865393ae40 P 598969e0661641f7839d6402bd26c612: Time spent bootstrapping tablet: real 0.026s user 0.017s sys 0.010s I0507 06:32:56.082651 13209 raft_consensus.cc:340] T 78673d8ea75d4380b30b00865393ae40 P 598969e0661641f7839d6402bd26c612 [term 1 LEARNER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: 40 OBSOLETE_local: false peers { permanent_uuid: "de47b12769c24f8489d00f790f475b8d" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 36909 } } peers { permanent_uuid: "e167493e37494c2896e3b5fe35a5a30d" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 37613 } } peers { permanent_uuid: "598969e0661641f7839d6402bd26c612" member_type: NON_VOTER last_known_addr { host: "127.0.57.194" port: 33509 } attrs { promote: true } } I0507 06:32:56.082904 13209 raft_consensus.cc:705] T 78673d8ea75d4380b30b00865393ae40 P 598969e0661641f7839d6402bd26c612 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 598969e0661641f7839d6402bd26c612, State: Initialized, Role: LEARNER I0507 06:32:56.083047 13209 consensus_queue.cc:229] T 78673d8ea75d4380b30b00865393ae40 P 598969e0661641f7839d6402bd26c612 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 40, Last appended: 1.40, Last appended by leader: 40, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 40 OBSOLETE_local: false peers { permanent_uuid: "de47b12769c24f8489d00f790f475b8d" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 36909 } } peers { permanent_uuid: "e167493e37494c2896e3b5fe35a5a30d" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 37613 } } peers { permanent_uuid: "598969e0661641f7839d6402bd26c612" member_type: NON_VOTER last_known_addr { host: "127.0.57.194" port: 33509 } attrs { promote: true } } I0507 06:32:56.083823 13209 ts_tablet_manager.cc:1081] T 78673d8ea75d4380b30b00865393ae40 P 598969e0661641f7839d6402bd26c612: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s I0507 06:32:56.084970 12769 tablet_copy_service.cc:325] P e167493e37494c2896e3b5fe35a5a30d: Request end of tablet copy session 598969e0661641f7839d6402bd26c612-78673d8ea75d4380b30b00865393ae40 received from {username='slave'} at 127.0.57.194:56019 I0507 06:32:56.085114 12769 tablet_copy_service.cc:413] P e167493e37494c2896e3b5fe35a5a30d: ending tablet copy session 598969e0661641f7839d6402bd26c612-78673d8ea75d4380b30b00865393ae40 on tablet 78673d8ea75d4380b30b00865393ae40 with peer 598969e0661641f7839d6402bd26c612 I0507 06:32:56.177711 12916 maintenance_manager.cc:306] P 598969e0661641f7839d6402bd26c612: Scheduling UndoDeltaBlockGCOp(78673d8ea75d4380b30b00865393ae40): 506 bytes on disk I0507 06:32:56.179374 12810 maintenance_manager.cc:501] P 598969e0661641f7839d6402bd26c612: UndoDeltaBlockGCOp(78673d8ea75d4380b30b00865393ae40) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":113,"lbm_reads_lt_1ms":4} I0507 06:32:56.521423 12872 raft_consensus.cc:1126] T 78673d8ea75d4380b30b00865393ae40 P 598969e0661641f7839d6402bd26c612 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.39->[1.40-1.40] Dedup: 1.40->[] I0507 06:32:57.123201 13213 raft_consensus.cc:974] T 78673d8ea75d4380b30b00865393ae40 P e167493e37494c2896e3b5fe35a5a30d: attempting to promote NON_VOTER 598969e0661641f7839d6402bd26c612 to VOTER I0507 06:32:57.123905 13213 consensus_queue.cc:206] T 78673d8ea75d4380b30b00865393ae40 P e167493e37494c2896e3b5fe35a5a30d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 40, Committed index: 40, Last appended: 1.40, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 41 OBSOLETE_local: false peers { permanent_uuid: "de47b12769c24f8489d00f790f475b8d" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 36909 } } peers { permanent_uuid: "e167493e37494c2896e3b5fe35a5a30d" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 37613 } } peers { permanent_uuid: "598969e0661641f7839d6402bd26c612" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 33509 } attrs { promote: false } } I0507 06:32:57.126094 12872 raft_consensus.cc:1181] T 78673d8ea75d4380b30b00865393ae40 P 598969e0661641f7839d6402bd26c612 [term 1 LEARNER]: Refusing update from remote peer e167493e37494c2896e3b5fe35a5a30d: Log matching property violated. Preceding OpId in replica: term: 1 index: 40. Preceding OpId from leader: term: 1 index: 41. (index mismatch) I0507 06:32:57.126586 13214 consensus_queue.cc:984] T 78673d8ea75d4380b30b00865393ae40 P e167493e37494c2896e3b5fe35a5a30d [LEADER]: Connected to new peer: Peer: permanent_uuid: "598969e0661641f7839d6402bd26c612" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 33509 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 41, Last known committed idx: 40, Time since last communication: 0.000s I0507 06:32:57.126698 12994 raft_consensus.cc:1181] T 78673d8ea75d4380b30b00865393ae40 P de47b12769c24f8489d00f790f475b8d [term 1 FOLLOWER]: Refusing update from remote peer e167493e37494c2896e3b5fe35a5a30d: Log matching property violated. Preceding OpId in replica: term: 1 index: 40. Preceding OpId from leader: term: 1 index: 41. (index mismatch) I0507 06:32:57.127183 13214 consensus_queue.cc:984] T 78673d8ea75d4380b30b00865393ae40 P e167493e37494c2896e3b5fe35a5a30d [LEADER]: Connected to new peer: Peer: permanent_uuid: "de47b12769c24f8489d00f790f475b8d" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 36909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 41, Last known committed idx: 40, Time since last communication: 0.000s I0507 06:32:57.129251 13213 raft_consensus.cc:2788] T 78673d8ea75d4380b30b00865393ae40 P e167493e37494c2896e3b5fe35a5a30d [term 1 LEADER]: Committing config change with OpId 1.41: config changed from index 40 to 41, 598969e0661641f7839d6402bd26c612 (127.0.57.194) changed from NON_VOTER to VOTER. New config: { opid_index: 41 OBSOLETE_local: false peers { permanent_uuid: "de47b12769c24f8489d00f790f475b8d" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 36909 } } peers { permanent_uuid: "e167493e37494c2896e3b5fe35a5a30d" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 37613 } } peers { permanent_uuid: "598969e0661641f7839d6402bd26c612" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 33509 } attrs { promote: false } } } I0507 06:32:57.129842 12872 raft_consensus.cc:2788] T 78673d8ea75d4380b30b00865393ae40 P 598969e0661641f7839d6402bd26c612 [term 1 FOLLOWER]: Committing config change with OpId 1.41: config changed from index 40 to 41, 598969e0661641f7839d6402bd26c612 (127.0.57.194) changed from NON_VOTER to VOTER. New config: { opid_index: 41 OBSOLETE_local: false peers { permanent_uuid: "de47b12769c24f8489d00f790f475b8d" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 36909 } } peers { permanent_uuid: "e167493e37494c2896e3b5fe35a5a30d" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 37613 } } peers { permanent_uuid: "598969e0661641f7839d6402bd26c612" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 33509 } attrs { promote: false } } } I0507 06:32:57.132797 12994 raft_consensus.cc:2788] T 78673d8ea75d4380b30b00865393ae40 P de47b12769c24f8489d00f790f475b8d [term 1 FOLLOWER]: Committing config change with OpId 1.41: config changed from index 40 to 41, 598969e0661641f7839d6402bd26c612 (127.0.57.194) changed from NON_VOTER to VOTER. New config: { opid_index: 41 OBSOLETE_local: false peers { permanent_uuid: "de47b12769c24f8489d00f790f475b8d" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 36909 } } peers { permanent_uuid: "e167493e37494c2896e3b5fe35a5a30d" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 37613 } } peers { permanent_uuid: "598969e0661641f7839d6402bd26c612" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 33509 } attrs { promote: false } } } I0507 06:32:57.133919 13146 catalog_manager.cc:3988] T 78673d8ea75d4380b30b00865393ae40 P e167493e37494c2896e3b5fe35a5a30d reported cstate change: config changed from index 40 to 41, 598969e0661641f7839d6402bd26c612 (127.0.57.194) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e167493e37494c2896e3b5fe35a5a30d" committed_config { opid_index: 41 OBSOLETE_local: false peers { permanent_uuid: "de47b12769c24f8489d00f790f475b8d" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 36909 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e167493e37494c2896e3b5fe35a5a30d" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 37613 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "598969e0661641f7839d6402bd26c612" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 33509 } attrs { promote: false } health_report { overall_health: HEALTHY } } } Master Summary UUID | Address | Status ----------------------------------+--------------------+--------- a1bfe1b2ebf24d0ca70399022578708a | 127.0.57.254:36417 | HEALTHY Flag | Value | Tags | Master ------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+------------------------- ipki_ca_key_size | 1024 | experimental | all 1 server(s) checked ipki_server_key_size | 1024 | experimental | all 1 server(s) checked master_add_server_when_underreplicated | true | hidden | all 1 server(s) checked never_fsync | true | unsafe,advanced | all 1 server(s) checked raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked rpc_reuseport | true | experimental | all 1 server(s) checked rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked server_dump_info_format | pb | hidden | all 1 server(s) checked server_dump_info_path | /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_0.1557210500371863-231/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked Tablet Server Summary UUID | Address | Status | Location ----------------------------------+--------------------+---------+---------- 598969e0661641f7839d6402bd26c612 | 127.0.57.194:33509 | HEALTHY | de47b12769c24f8489d00f790f475b8d | 127.0.57.195:36909 | HEALTHY | e167493e37494c2896e3b5fe35a5a30d | 127.0.57.193:37613 | HEALTHY | Tablet Server Location Summary Location | Count ----------+--------- | 3 Flag | Value | Tags | Tablet Server ------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------------------------------+------------------------- ipki_server_key_size | 1024 | experimental | all 3 server(s) checked local_ip_for_outbound_sockets | 127.0.57.193 | experimental | 127.0.57.193:37613 local_ip_for_outbound_sockets | 127.0.57.194 | experimental | 127.0.57.194:33509 local_ip_for_outbound_sockets | 127.0.57.195 | experimental | 127.0.57.195:36909 log_compression_codec | none | experimental | all 3 server(s) checked log_max_segments_to_retain | 3 | experimental,advanced,runtime | all 3 server(s) checked log_target_replay_size_mb | 1 | experimental | all 3 server(s) checked maintenance_manager_polling_interval_ms | 100 | hidden | all 3 server(s) checked max_cell_size_bytes | 1000000 | unsafe | all 3 server(s) checked never_fsync | true | unsafe,advanced | all 3 server(s) checked raft_prepare_replacement_before_eviction | true | experimental,advanced | all 3 server(s) checked rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked server_dump_info_format | pb | hidden | all 3 server(s) checked server_dump_info_path | /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_0.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.0.57.193:37613 server_dump_info_path | /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_0.1557210500371863-231/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.0.57.194:33509 server_dump_info_path | /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_0.1557210500371863-231/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.0.57.195:36909 Version Summary Version | Servers ------------------------+------------------------- 1.10.0-cdh6.x-SNAPSHOT | all 4 server(s) checked Tablet Summary Summary by table Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable -----------+----+---------+---------------+---------+------------+------------------+------------- TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0 Tablet Replica Count Summary Statistic | Replica Count ----------------+--------------- Minimum | 1 First Quartile | 1 Median | 1 Third Quartile | 1 Maximum | 1 Total Count Summary | Total Count ----------------+------------- Masters | 1 Tablet Servers | 3 Tables | 1 Tablets | 1 Replicas | 3 ================== Warnings: ================== Some masters have unsafe, experimental, or hidden flags set Some tablet servers have unsafe, experimental, or hidden flags set OK I0507 06:32:57.234696 231 log_verifier.cc:121] Checking tablet 78673d8ea75d4380b30b00865393ae40 I0507 06:32:57.254359 231 log_verifier.cc:172] Verified matching terms for 14 ops in tablet 78673d8ea75d4380b30b00865393ae40 I0507 06:32:57.255702 231 external_mini_cluster.cc:1172] Attempting to check leaks for /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver pid 12663 I0507 06:32:57.257040 12789 generic_service.cc:171] Checking for leaks (request via RPC) I0507 06:32:57.360636 231 external_mini_cluster.cc:1131] Killing /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver with pid 12663 I0507 06:32:57.377890 231 external_mini_cluster.cc:1172] Attempting to check leaks for /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver pid 12796 I0507 06:32:57.378898 12912 generic_service.cc:171] Checking for leaks (request via RPC) I0507 06:32:57.435578 231 external_mini_cluster.cc:1131] Killing /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver with pid 12796 I0507 06:32:57.447075 231 external_mini_cluster.cc:1172] Attempting to check leaks for /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver pid 12918 I0507 06:32:57.448179 13034 generic_service.cc:171] Checking for leaks (request via RPC) I0507 06:32:57.539927 231 external_mini_cluster.cc:1131] Killing /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver with pid 12918 I0507 06:32:57.555323 231 external_mini_cluster.cc:1172] Attempting to check leaks for /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-master pid 13121 I0507 06:32:57.556517 13177 generic_service.cc:171] Checking for leaks (request via RPC) I0507 06:32:57.661504 231 external_mini_cluster.cc:1131] Killing /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-master with pid 13121 [ OK ] ReplicaBehindWalGcThresholdITest.ReplicaReplacement/0 (7692 ms) [ RUN ] ReplicaBehindWalGcThresholdITest.ReplicaReplacement/1 I0507 06:32:57.684420 231 test_util.cc:212] Using random seed: 1665042850 I0507 06:32:57.685930 231 ts_itest-base.cc:112] Starting cluster with: I0507 06:32:57.685982 231 ts_itest-base.cc:113] -------------- I0507 06:32:57.686003 231 ts_itest-base.cc:114] 3 tablet servers I0507 06:32:57.686048 231 ts_itest-base.cc:115] 3 replicas per TS I0507 06:32:57.686071 231 ts_itest-base.cc:116] -------------- I0507 06:32:57.687207 231 external_mini_cluster.cc:933] Running /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-master /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-master --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_1.1557210500371863-231/raft_consensus-itest-cluster/master-0/wal --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_1.1557210500371863-231/raft_consensus-itest-cluster/master-0/data --block_manager=log --webserver_interface=localhost --ipki_ca_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.0.57.254:37965 --webserver_interface=127.0.57.254 --webserver_port=0 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_1.1557210500371863-231/raft_consensus-itest-cluster/master-0/logs --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_1.1557210500371863-231/raft_consensus-itest-cluster/master-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=true --master_add_server_when_underreplicated=false --rpc_reuseport=true with env {} WARNING: Logging before InitGoogleLogging() is written to STDERR W0507 06:32:57.900169 13260 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0507 06:32:57.900400 13260 flags.cc:404] Enabled unsafe flag: --never_fsync=true W0507 06:32:57.902914 13260 flags.cc:404] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true W0507 06:32:57.902987 13260 flags.cc:404] Enabled experimental flag: --ipki_ca_key_size=1024 W0507 06:32:57.903017 13260 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W0507 06:32:57.903051 13260 flags.cc:404] Enabled experimental flag: --tsk_num_rsa_bits=512 W0507 06:32:57.903082 13260 flags.cc:404] Enabled experimental flag: --rpc_reuseport=true I0507 06:32:57.907938 13260 master_main.cc:73] Master server non-default flags: --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_1.1557210500371863-231/raft_consensus-itest-cluster/master-0/data --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_1.1557210500371863-231/raft_consensus-itest-cluster/master-0/wal --master_add_server_when_underreplicated=false --ipki_ca_key_size=1024 --ipki_server_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.0.57.254:37965 --rpc_reuseport=true --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_1.1557210500371863-231/raft_consensus-itest-cluster/master-0/data/info.pb --webserver_interface=127.0.57.254 --webserver_port=0 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_1.1557210500371863-231/raft_consensus-itest-cluster/master-0/logs --logbuflevel=-1 --logtostderr=true Master server version: kudu 1.10.0-cdh6.x-SNAPSHOT revision 7d0ee26697926e87af9ef395008799577f6ab43f build type FASTDEBUG built by jenkins at 06 May 2019 22:44:21 PST on quasar-knmpfx-1.vpc.cloudera.com build id 322017 ASAN enabled I0507 06:32:57.914876 13260 system_ntp.cc:202] NTP initialized. Skew: 500ppm Current error: 163016us I0507 06:32:57.916141 13260 fs_manager.cc:263] Metadata directory not provided I0507 06:32:57.916210 13260 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0507 06:32:57.916479 13260 server_base.cc:433] Could not load existing FS layout: Not found: could not find a healthy instance file I0507 06:32:57.916568 13260 server_base.cc:434] Attempting to create new FS layout instead I0507 06:32:57.919061 13260 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_1.1557210500371863-231/raft_consensus-itest-cluster/master-0/data/instance: uuid: "0c8516a734564649b36e53a7f5184239" format_stamp: "Formatted at 2019-05-07 06:32:57 on dist-test-slave-dist-test-slave-rqp6" I0507 06:32:57.919572 13260 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_1.1557210500371863-231/raft_consensus-itest-cluster/master-0/wal/instance: uuid: "0c8516a734564649b36e53a7f5184239" format_stamp: "Formatted at 2019-05-07 06:32:57 on dist-test-slave-dist-test-slave-rqp6" I0507 06:32:57.926190 13260 fs_manager.cc:503] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s I0507 06:32:57.927245 13260 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0507 06:32:57.927817 13260 file_cache.cc:466] Constructed file cache lbm with capacity 419430 I0507 06:32:57.931473 13260 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s I0507 06:32:57.931563 13260 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_1.1557210500371863-231/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_1.1557210500371863-231/raft_consensus-itest-cluster/master-0/wal uuid: "0c8516a734564649b36e53a7f5184239" format_stamp: "Formatted at 2019-05-07 06:32:57 on dist-test-slave-dist-test-slave-rqp6" I0507 06:32:57.931710 13260 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_1.1557210500371863-231/raft_consensus-itest-cluster/master-0/wal metadata directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_1.1557210500371863-231/raft_consensus-itest-cluster/master-0/wal 1 data directories: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_1.1557210500371863-231/raft_consensus-itest-cluster/master-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0507 06:32:58.013510 13260 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0507 06:32:58.031450 13260 rpc_server.cc:205] RPC server started. Bound to: 127.0.57.254:37965 I0507 06:32:58.032907 13260 webserver.cc:308] Webserver started at http://127.0.57.254:40901/ using document root and password file I0507 06:32:58.033814 13260 server_base.cc:608] Dumped server information to /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_1.1557210500371863-231/raft_consensus-itest-cluster/master-0/data/info.pb I0507 06:32:58.037859 13319 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0507 06:32:58.038854 231 external_mini_cluster.cc:995] Started /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-master as pid 13260 I0507 06:32:58.039553 231 external_mini_cluster.cc:933] Running /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_1.1557210500371863-231/raft_consensus-itest-cluster/ts-0/wal --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_1.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data --block_manager=log --rpc_bind_addresses=127.0.57.193:0 --local_ip_for_outbound_sockets=127.0.57.193 --webserver_interface=127.0.57.193 --webserver_port=0 --tserver_master_addrs=127.0.57.254:37965 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_1.1557210500371863-231/raft_consensus-itest-cluster/ts-0/logs --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_1.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=true --follower_unavailable_considered_failed_sec=5 --log_compression_codec=none --log_cache_size_limit_mb=1 --log_segment_size_mb=1 --log_async_preallocate_segments=false --log_min_segments_to_retain=1 --log_max_segments_to_retain=3 --maintenance_manager_polling_interval_ms=100 --log_target_replay_size_mb=1 --max_cell_size_bytes=1000000 with env {} I0507 06:32:58.050117 13319 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 0c8516a734564649b36e53a7f5184239: Bootstrap starting. I0507 06:32:58.057458 13319 tablet_bootstrap.cc:590] T 00000000000000000000000000000000 P 0c8516a734564649b36e53a7f5184239: No blocks or log segments found. Creating new log. I0507 06:32:58.058557 13319 log.cc:525] T 00000000000000000000000000000000 P 0c8516a734564649b36e53a7f5184239: Log is configured to *not* fsync() on all Append() calls I0507 06:32:58.066529 13319 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 0c8516a734564649b36e53a7f5184239: No bootstrap required, opened a new log I0507 06:32:58.074133 13319 raft_consensus.cc:340] T 00000000000000000000000000000000 P 0c8516a734564649b36e53a7f5184239 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c8516a734564649b36e53a7f5184239" member_type: VOTER } I0507 06:32:58.074405 13319 raft_consensus.cc:366] T 00000000000000000000000000000000 P 0c8516a734564649b36e53a7f5184239 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0507 06:32:58.074460 13319 raft_consensus.cc:705] T 00000000000000000000000000000000 P 0c8516a734564649b36e53a7f5184239 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c8516a734564649b36e53a7f5184239, State: Initialized, Role: FOLLOWER I0507 06:32:58.075139 13319 consensus_queue.cc:229] T 00000000000000000000000000000000 P 0c8516a734564649b36e53a7f5184239 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c8516a734564649b36e53a7f5184239" member_type: VOTER } I0507 06:32:58.075318 13319 raft_consensus.cc:380] T 00000000000000000000000000000000 P 0c8516a734564649b36e53a7f5184239 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0507 06:32:58.075402 13319 raft_consensus.cc:472] T 00000000000000000000000000000000 P 0c8516a734564649b36e53a7f5184239 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0507 06:32:58.075547 13319 raft_consensus.cc:2886] T 00000000000000000000000000000000 P 0c8516a734564649b36e53a7f5184239 [term 0 FOLLOWER]: Advancing to term 1 I0507 06:32:58.077572 13319 raft_consensus.cc:494] T 00000000000000000000000000000000 P 0c8516a734564649b36e53a7f5184239 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c8516a734564649b36e53a7f5184239" member_type: VOTER } I0507 06:32:58.078054 13319 leader_election.cc:310] T 00000000000000000000000000000000 P 0c8516a734564649b36e53a7f5184239 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0c8516a734564649b36e53a7f5184239; no voters: I0507 06:32:58.078982 13319 leader_election.cc:296] T 00000000000000000000000000000000 P 0c8516a734564649b36e53a7f5184239 [CANDIDATE]: Term 1 election: Requested vote from peers I0507 06:32:58.079320 13322 raft_consensus.cc:2640] T 00000000000000000000000000000000 P 0c8516a734564649b36e53a7f5184239 [term 1 FOLLOWER]: Leader election won for term 1 I0507 06:32:58.080137 13322 raft_consensus.cc:667] T 00000000000000000000000000000000 P 0c8516a734564649b36e53a7f5184239 [term 1 LEADER]: Becoming Leader. State: Replica: 0c8516a734564649b36e53a7f5184239, State: Running, Role: LEADER I0507 06:32:58.080723 13322 consensus_queue.cc:206] T 00000000000000000000000000000000 P 0c8516a734564649b36e53a7f5184239 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c8516a734564649b36e53a7f5184239" member_type: VOTER } I0507 06:32:58.084985 13324 sys_catalog.cc:337] T 00000000000000000000000000000000 P 0c8516a734564649b36e53a7f5184239 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0c8516a734564649b36e53a7f5184239" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c8516a734564649b36e53a7f5184239" member_type: VOTER } } I0507 06:32:58.084803 13323 sys_catalog.cc:337] T 00000000000000000000000000000000 P 0c8516a734564649b36e53a7f5184239 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0c8516a734564649b36e53a7f5184239. Latest consensus state: current_term: 1 leader_uuid: "0c8516a734564649b36e53a7f5184239" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c8516a734564649b36e53a7f5184239" member_type: VOTER } } I0507 06:32:58.085949 13319 sys_catalog.cc:337] T 00000000000000000000000000000000 P 0c8516a734564649b36e53a7f5184239 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 1 leader_uuid: "0c8516a734564649b36e53a7f5184239" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c8516a734564649b36e53a7f5184239" member_type: VOTER } } I0507 06:32:58.086055 13323 sys_catalog.cc:340] T 00000000000000000000000000000000 P 0c8516a734564649b36e53a7f5184239 [sys.catalog]: This master's current role is: LEADER I0507 06:32:58.086107 13319 sys_catalog.cc:340] T 00000000000000000000000000000000 P 0c8516a734564649b36e53a7f5184239 [sys.catalog]: This master's current role is: LEADER I0507 06:32:58.085952 13324 sys_catalog.cc:340] T 00000000000000000000000000000000 P 0c8516a734564649b36e53a7f5184239 [sys.catalog]: This master's current role is: LEADER I0507 06:32:58.087718 13319 sys_catalog.cc:424] T 00000000000000000000000000000000 P 0c8516a734564649b36e53a7f5184239 [sys.catalog]: configured and running, proceeding with master startup. I0507 06:32:58.088274 13328 catalog_manager.cc:1055] Loading table and tablet metadata into memory... I0507 06:32:58.093322 13328 catalog_manager.cc:1066] Initializing Kudu internal certificate authority... I0507 06:32:58.129789 13328 catalog_manager.cc:950] Generated new certificate authority record I0507 06:32:58.132025 13328 catalog_manager.cc:1075] Loading token signing keys... I0507 06:32:58.144598 13328 catalog_manager.cc:4292] T 00000000000000000000000000000000 P 0c8516a734564649b36e53a7f5184239: Generated new TSK 0 WARNING: Logging before InitGoogleLogging() is written to STDERR W0507 06:32:58.227229 13320 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0507 06:32:58.227488 13320 flags.cc:404] Enabled unsafe flag: --max_cell_size_bytes=1000000 W0507 06:32:58.227551 13320 flags.cc:404] Enabled unsafe flag: --never_fsync=true W0507 06:32:58.229804 13320 flags.cc:404] Enabled experimental flag: --log_compression_codec=none W0507 06:32:58.229867 13320 flags.cc:404] Enabled experimental flag: --log_max_segments_to_retain=3 W0507 06:32:58.229933 13320 flags.cc:404] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true W0507 06:32:58.229996 13320 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W0507 06:32:58.230121 13320 flags.cc:404] Enabled experimental flag: --log_target_replay_size_mb=1 W0507 06:32:58.230183 13320 flags.cc:404] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.57.193 I0507 06:32:58.237363 13320 tablet_server_main.cc:78] Tablet server non-default flags: --follower_unavailable_considered_failed_sec=5 --log_compression_codec=none --log_max_segments_to_retain=3 --log_cache_size_limit_mb=1 --log_async_preallocate_segments=false --log_segment_size_mb=1 --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_1.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_1.1557210500371863-231/raft_consensus-itest-cluster/ts-0/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.0.57.193:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_1.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data/info.pb --webserver_interface=127.0.57.193 --webserver_port=0 --max_cell_size_bytes=1000000 --tserver_master_addrs=127.0.57.254:37965 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --log_target_replay_size_mb=1 --maintenance_manager_polling_interval_ms=100 --enable_minidumps=false --local_ip_for_outbound_sockets=127.0.57.193 --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_1.1557210500371863-231/raft_consensus-itest-cluster/ts-0/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.10.0-cdh6.x-SNAPSHOT revision 7d0ee26697926e87af9ef395008799577f6ab43f build type FASTDEBUG built by jenkins at 06 May 2019 22:44:21 PST on quasar-knmpfx-1.vpc.cloudera.com build id 322017 ASAN enabled I0507 06:32:58.245654 13320 system_ntp.cc:202] NTP initialized. Skew: 500ppm Current error: 163516us I0507 06:32:58.246394 13320 fs_manager.cc:263] Metadata directory not provided I0507 06:32:58.246460 13320 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0507 06:32:58.246804 13320 server_base.cc:433] Could not load existing FS layout: Not found: could not find a healthy instance file I0507 06:32:58.246873 13320 server_base.cc:434] Attempting to create new FS layout instead I0507 06:32:58.249593 13320 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_1.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data/instance: uuid: "89dd992cec6b407ca4e21db5e6aeef70" format_stamp: "Formatted at 2019-05-07 06:32:58 on dist-test-slave-dist-test-slave-rqp6" I0507 06:32:58.250237 13320 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_1.1557210500371863-231/raft_consensus-itest-cluster/ts-0/wal/instance: uuid: "89dd992cec6b407ca4e21db5e6aeef70" format_stamp: "Formatted at 2019-05-07 06:32:58 on dist-test-slave-dist-test-slave-rqp6" I0507 06:32:58.259325 13320 fs_manager.cc:503] Time spent creating directory manager: real 0.009s user 0.005s sys 0.004s I0507 06:32:58.260479 13320 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0507 06:32:58.261198 13320 file_cache.cc:466] Constructed file cache lbm with capacity 419430 I0507 06:32:58.264361 13320 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.000s sys 0.000s I0507 06:32:58.264452 13320 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_1.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_1.1557210500371863-231/raft_consensus-itest-cluster/ts-0/wal uuid: "89dd992cec6b407ca4e21db5e6aeef70" format_stamp: "Formatted at 2019-05-07 06:32:58 on dist-test-slave-dist-test-slave-rqp6" I0507 06:32:58.264739 13320 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_1.1557210500371863-231/raft_consensus-itest-cluster/ts-0/wal metadata directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_1.1557210500371863-231/raft_consensus-itest-cluster/ts-0/wal 1 data directories: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_1.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0507 06:32:58.306418 13320 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0507 06:32:58.308293 13320 ts_tablet_manager.cc:356] Loaded tablet metadata (0 total tablets, 0 live tablets) I0507 06:32:58.308382 13320 ts_tablet_manager.cc:376] Registered 0 tablets I0507 06:32:58.405337 13320 rpc_server.cc:205] RPC server started. Bound to: 127.0.57.193:40533 I0507 06:32:58.406687 13320 webserver.cc:308] Webserver started at http://127.0.57.193:43315/ using document root and password file I0507 06:32:58.407477 13320 server_base.cc:608] Dumped server information to /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_1.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data/info.pb I0507 06:32:58.410773 231 external_mini_cluster.cc:995] Started /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver as pid 13320 I0507 06:32:58.411409 231 external_mini_cluster.cc:933] Running /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_1.1557210500371863-231/raft_consensus-itest-cluster/ts-1/wal --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_1.1557210500371863-231/raft_consensus-itest-cluster/ts-1/data --block_manager=log --rpc_bind_addresses=127.0.57.194:0 --local_ip_for_outbound_sockets=127.0.57.194 --webserver_interface=127.0.57.194 --webserver_port=0 --tserver_master_addrs=127.0.57.254:37965 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_1.1557210500371863-231/raft_consensus-itest-cluster/ts-1/logs --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_1.1557210500371863-231/raft_consensus-itest-cluster/ts-1/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=true --follower_unavailable_considered_failed_sec=5 --log_compression_codec=none --log_cache_size_limit_mb=1 --log_segment_size_mb=1 --log_async_preallocate_segments=false --log_min_segments_to_retain=1 --log_max_segments_to_retain=3 --maintenance_manager_polling_interval_ms=100 --log_target_replay_size_mb=1 --max_cell_size_bytes=1000000 with env {} I0507 06:32:58.436674 13450 heartbeater.cc:345] Connected to a master server at 127.0.57.254:37965 I0507 06:32:58.436908 13450 heartbeater.cc:426] Registering TS with master... I0507 06:32:58.437532 13450 heartbeater.cc:475] Master 127.0.57.254:37965 requested a full tablet report, sending... I0507 06:32:58.439599 13286 ts_manager.cc:159] Registered new tserver with Master: 89dd992cec6b407ca4e21db5e6aeef70 (127.0.57.193:40533) I0507 06:32:58.441970 13286 master_service.cc:273] Signed X509 certificate for tserver {username='slave'} at 127.0.57.193:57455 WARNING: Logging before InitGoogleLogging() is written to STDERR W0507 06:32:58.594744 13454 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0507 06:32:58.594969 13454 flags.cc:404] Enabled unsafe flag: --max_cell_size_bytes=1000000 W0507 06:32:58.595017 13454 flags.cc:404] Enabled unsafe flag: --never_fsync=true W0507 06:32:58.597153 13454 flags.cc:404] Enabled experimental flag: --log_compression_codec=none W0507 06:32:58.597226 13454 flags.cc:404] Enabled experimental flag: --log_max_segments_to_retain=3 W0507 06:32:58.597273 13454 flags.cc:404] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true W0507 06:32:58.597328 13454 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W0507 06:32:58.597424 13454 flags.cc:404] Enabled experimental flag: --log_target_replay_size_mb=1 W0507 06:32:58.597470 13454 flags.cc:404] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.57.194 I0507 06:32:58.601914 13454 tablet_server_main.cc:78] Tablet server non-default flags: --follower_unavailable_considered_failed_sec=5 --log_compression_codec=none --log_max_segments_to_retain=3 --log_cache_size_limit_mb=1 --log_async_preallocate_segments=false --log_segment_size_mb=1 --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_1.1557210500371863-231/raft_consensus-itest-cluster/ts-1/data --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_1.1557210500371863-231/raft_consensus-itest-cluster/ts-1/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.0.57.194:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_1.1557210500371863-231/raft_consensus-itest-cluster/ts-1/data/info.pb --webserver_interface=127.0.57.194 --webserver_port=0 --max_cell_size_bytes=1000000 --tserver_master_addrs=127.0.57.254:37965 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --log_target_replay_size_mb=1 --maintenance_manager_polling_interval_ms=100 --enable_minidumps=false --local_ip_for_outbound_sockets=127.0.57.194 --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_1.1557210500371863-231/raft_consensus-itest-cluster/ts-1/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.10.0-cdh6.x-SNAPSHOT revision 7d0ee26697926e87af9ef395008799577f6ab43f build type FASTDEBUG built by jenkins at 06 May 2019 22:44:21 PST on quasar-knmpfx-1.vpc.cloudera.com build id 322017 ASAN enabled I0507 06:32:58.610525 13454 system_ntp.cc:202] NTP initialized. Skew: 500ppm Current error: 163516us I0507 06:32:58.611248 13454 fs_manager.cc:263] Metadata directory not provided I0507 06:32:58.611320 13454 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0507 06:32:58.611604 13454 server_base.cc:433] Could not load existing FS layout: Not found: could not find a healthy instance file I0507 06:32:58.611678 13454 server_base.cc:434] Attempting to create new FS layout instead I0507 06:32:58.614250 13454 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_1.1557210500371863-231/raft_consensus-itest-cluster/ts-1/data/instance: uuid: "86eeb3337bd640599d56daee2302ae67" format_stamp: "Formatted at 2019-05-07 06:32:58 on dist-test-slave-dist-test-slave-rqp6" I0507 06:32:58.614938 13454 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_1.1557210500371863-231/raft_consensus-itest-cluster/ts-1/wal/instance: uuid: "86eeb3337bd640599d56daee2302ae67" format_stamp: "Formatted at 2019-05-07 06:32:58 on dist-test-slave-dist-test-slave-rqp6" I0507 06:32:58.621702 13454 fs_manager.cc:503] Time spent creating directory manager: real 0.007s user 0.005s sys 0.000s I0507 06:32:58.622745 13454 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0507 06:32:58.623333 13454 file_cache.cc:466] Constructed file cache lbm with capacity 419430 I0507 06:32:58.626780 13454 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s I0507 06:32:58.626888 13454 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_1.1557210500371863-231/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_1.1557210500371863-231/raft_consensus-itest-cluster/ts-1/wal uuid: "86eeb3337bd640599d56daee2302ae67" format_stamp: "Formatted at 2019-05-07 06:32:58 on dist-test-slave-dist-test-slave-rqp6" I0507 06:32:58.627055 13454 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_1.1557210500371863-231/raft_consensus-itest-cluster/ts-1/wal metadata directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_1.1557210500371863-231/raft_consensus-itest-cluster/ts-1/wal 1 data directories: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_1.1557210500371863-231/raft_consensus-itest-cluster/ts-1/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0507 06:32:58.695616 13454 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0507 06:32:58.697511 13454 ts_tablet_manager.cc:356] Loaded tablet metadata (0 total tablets, 0 live tablets) I0507 06:32:58.697592 13454 ts_tablet_manager.cc:376] Registered 0 tablets I0507 06:32:58.726240 13454 rpc_server.cc:205] RPC server started. Bound to: 127.0.57.194:37763 I0507 06:32:58.727644 13454 webserver.cc:308] Webserver started at http://127.0.57.194:35307/ using document root and password file I0507 06:32:58.728494 13454 server_base.cc:608] Dumped server information to /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_1.1557210500371863-231/raft_consensus-itest-cluster/ts-1/data/info.pb I0507 06:32:58.729665 231 external_mini_cluster.cc:995] Started /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver as pid 13454 I0507 06:32:58.730338 231 external_mini_cluster.cc:933] Running /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_1.1557210500371863-231/raft_consensus-itest-cluster/ts-2/wal --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_1.1557210500371863-231/raft_consensus-itest-cluster/ts-2/data --block_manager=log --rpc_bind_addresses=127.0.57.195:0 --local_ip_for_outbound_sockets=127.0.57.195 --webserver_interface=127.0.57.195 --webserver_port=0 --tserver_master_addrs=127.0.57.254:37965 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_1.1557210500371863-231/raft_consensus-itest-cluster/ts-2/logs --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_1.1557210500371863-231/raft_consensus-itest-cluster/ts-2/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=true --follower_unavailable_considered_failed_sec=5 --log_compression_codec=none --log_cache_size_limit_mb=1 --log_segment_size_mb=1 --log_async_preallocate_segments=false --log_min_segments_to_retain=1 --log_max_segments_to_retain=3 --maintenance_manager_polling_interval_ms=100 --log_target_replay_size_mb=1 --max_cell_size_bytes=1000000 with env {} I0507 06:32:58.747145 13573 heartbeater.cc:345] Connected to a master server at 127.0.57.254:37965 I0507 06:32:58.747340 13573 heartbeater.cc:426] Registering TS with master... I0507 06:32:58.747874 13573 heartbeater.cc:475] Master 127.0.57.254:37965 requested a full tablet report, sending... I0507 06:32:58.749044 13286 ts_manager.cc:159] Registered new tserver with Master: 86eeb3337bd640599d56daee2302ae67 (127.0.57.194:37763) I0507 06:32:58.750425 13286 master_service.cc:273] Signed X509 certificate for tserver {username='slave'} at 127.0.57.194:41911 WARNING: Logging before InitGoogleLogging() is written to STDERR W0507 06:32:58.905774 13576 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0507 06:32:58.906008 13576 flags.cc:404] Enabled unsafe flag: --max_cell_size_bytes=1000000 W0507 06:32:58.906054 13576 flags.cc:404] Enabled unsafe flag: --never_fsync=true W0507 06:32:58.908120 13576 flags.cc:404] Enabled experimental flag: --log_compression_codec=none W0507 06:32:58.908182 13576 flags.cc:404] Enabled experimental flag: --log_max_segments_to_retain=3 W0507 06:32:58.908227 13576 flags.cc:404] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true W0507 06:32:58.908277 13576 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W0507 06:32:58.908360 13576 flags.cc:404] Enabled experimental flag: --log_target_replay_size_mb=1 W0507 06:32:58.908412 13576 flags.cc:404] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.57.195 I0507 06:32:58.913218 13576 tablet_server_main.cc:78] Tablet server non-default flags: --follower_unavailable_considered_failed_sec=5 --log_compression_codec=none --log_max_segments_to_retain=3 --log_cache_size_limit_mb=1 --log_async_preallocate_segments=false --log_segment_size_mb=1 --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_1.1557210500371863-231/raft_consensus-itest-cluster/ts-2/data --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_1.1557210500371863-231/raft_consensus-itest-cluster/ts-2/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.0.57.195:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_1.1557210500371863-231/raft_consensus-itest-cluster/ts-2/data/info.pb --webserver_interface=127.0.57.195 --webserver_port=0 --max_cell_size_bytes=1000000 --tserver_master_addrs=127.0.57.254:37965 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --log_target_replay_size_mb=1 --maintenance_manager_polling_interval_ms=100 --enable_minidumps=false --local_ip_for_outbound_sockets=127.0.57.195 --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_1.1557210500371863-231/raft_consensus-itest-cluster/ts-2/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.10.0-cdh6.x-SNAPSHOT revision 7d0ee26697926e87af9ef395008799577f6ab43f build type FASTDEBUG built by jenkins at 06 May 2019 22:44:21 PST on quasar-knmpfx-1.vpc.cloudera.com build id 322017 ASAN enabled I0507 06:32:58.921214 13576 system_ntp.cc:202] NTP initialized. Skew: 500ppm Current error: 163516us I0507 06:32:58.921933 13576 fs_manager.cc:263] Metadata directory not provided I0507 06:32:58.921996 13576 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0507 06:32:58.922274 13576 server_base.cc:433] Could not load existing FS layout: Not found: could not find a healthy instance file I0507 06:32:58.922341 13576 server_base.cc:434] Attempting to create new FS layout instead I0507 06:32:58.924911 13576 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_1.1557210500371863-231/raft_consensus-itest-cluster/ts-2/data/instance: uuid: "b8748b85a6fb4418a992adbdfb1a0a62" format_stamp: "Formatted at 2019-05-07 06:32:58 on dist-test-slave-dist-test-slave-rqp6" I0507 06:32:58.925474 13576 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_1.1557210500371863-231/raft_consensus-itest-cluster/ts-2/wal/instance: uuid: "b8748b85a6fb4418a992adbdfb1a0a62" format_stamp: "Formatted at 2019-05-07 06:32:58 on dist-test-slave-dist-test-slave-rqp6" I0507 06:32:58.932596 13576 fs_manager.cc:503] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s I0507 06:32:58.933735 13576 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0507 06:32:58.934341 13576 file_cache.cc:466] Constructed file cache lbm with capacity 419430 I0507 06:32:58.937727 13576 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s I0507 06:32:58.937821 13576 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_1.1557210500371863-231/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_1.1557210500371863-231/raft_consensus-itest-cluster/ts-2/wal uuid: "b8748b85a6fb4418a992adbdfb1a0a62" format_stamp: "Formatted at 2019-05-07 06:32:58 on dist-test-slave-dist-test-slave-rqp6" I0507 06:32:58.938006 13576 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_1.1557210500371863-231/raft_consensus-itest-cluster/ts-2/wal metadata directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_1.1557210500371863-231/raft_consensus-itest-cluster/ts-2/wal 1 data directories: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_1.1557210500371863-231/raft_consensus-itest-cluster/ts-2/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0507 06:32:58.979620 13576 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0507 06:32:58.981632 13576 ts_tablet_manager.cc:356] Loaded tablet metadata (0 total tablets, 0 live tablets) I0507 06:32:58.981715 13576 ts_tablet_manager.cc:376] Registered 0 tablets I0507 06:32:59.008262 13576 rpc_server.cc:205] RPC server started. Bound to: 127.0.57.195:37113 I0507 06:32:59.009549 13576 webserver.cc:308] Webserver started at http://127.0.57.195:36863/ using document root and password file I0507 06:32:59.010370 13576 server_base.cc:608] Dumped server information to /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_1.1557210500371863-231/raft_consensus-itest-cluster/ts-2/data/info.pb I0507 06:32:59.012148 231 external_mini_cluster.cc:995] Started /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver as pid 13576 I0507 06:32:59.034327 13695 heartbeater.cc:345] Connected to a master server at 127.0.57.254:37965 I0507 06:32:59.034585 13695 heartbeater.cc:426] Registering TS with master... I0507 06:32:59.035130 13695 heartbeater.cc:475] Master 127.0.57.254:37965 requested a full tablet report, sending... I0507 06:32:59.036558 13285 ts_manager.cc:159] Registered new tserver with Master: b8748b85a6fb4418a992adbdfb1a0a62 (127.0.57.195:37113) I0507 06:32:59.037708 13285 master_service.cc:273] Signed X509 certificate for tserver {username='slave'} at 127.0.57.195:37609 I0507 06:32:59.038707 231 external_mini_cluster.cc:623] 3 TS(s) registered with all masters I0507 06:32:59.050586 13286 catalog_manager.cc:1415] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49246: name: "TestTable" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 3 split_rows_range_bounds { } partition_schema { range_schema { columns { name: "key" } } } W0507 06:32:59.055626 13286 catalog_manager.cc:1574] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestTable in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers. I0507 06:32:59.085003 13632 tablet_service.cc:1016] Processing CreateTablet for tablet d3b2287a051d4a8881adb4756316aaa1 (table=TestTable [id=26087773aad9435a9eef387b8101db47]), partition=RANGE (key) PARTITION UNBOUNDED I0507 06:32:59.085032 13387 tablet_service.cc:1016] Processing CreateTablet for tablet d3b2287a051d4a8881adb4756316aaa1 (table=TestTable [id=26087773aad9435a9eef387b8101db47]), partition=RANGE (key) PARTITION UNBOUNDED I0507 06:32:59.085541 13510 tablet_service.cc:1016] Processing CreateTablet for tablet d3b2287a051d4a8881adb4756316aaa1 (table=TestTable [id=26087773aad9435a9eef387b8101db47]), partition=RANGE (key) PARTITION UNBOUNDED I0507 06:32:59.086127 13632 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet d3b2287a051d4a8881adb4756316aaa1. 1 dirs total, 0 dirs full, 0 dirs failed I0507 06:32:59.086194 13387 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet d3b2287a051d4a8881adb4756316aaa1. 1 dirs total, 0 dirs full, 0 dirs failed I0507 06:32:59.086329 13510 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet d3b2287a051d4a8881adb4756316aaa1. 1 dirs total, 0 dirs full, 0 dirs failed I0507 06:32:59.097446 13714 tablet_bootstrap.cc:439] T d3b2287a051d4a8881adb4756316aaa1 P b8748b85a6fb4418a992adbdfb1a0a62: Bootstrap starting. I0507 06:32:59.103292 13715 tablet_bootstrap.cc:439] T d3b2287a051d4a8881adb4756316aaa1 P 86eeb3337bd640599d56daee2302ae67: Bootstrap starting. I0507 06:32:59.105950 13715 tablet_bootstrap.cc:590] T d3b2287a051d4a8881adb4756316aaa1 P 86eeb3337bd640599d56daee2302ae67: No blocks or log segments found. Creating new log. I0507 06:32:59.106945 13715 log.cc:525] T d3b2287a051d4a8881adb4756316aaa1 P 86eeb3337bd640599d56daee2302ae67: Log is configured to *not* fsync() on all Append() calls I0507 06:32:59.107722 13716 tablet_bootstrap.cc:439] T d3b2287a051d4a8881adb4756316aaa1 P 89dd992cec6b407ca4e21db5e6aeef70: Bootstrap starting. I0507 06:32:59.110790 13716 tablet_bootstrap.cc:590] T d3b2287a051d4a8881adb4756316aaa1 P 89dd992cec6b407ca4e21db5e6aeef70: No blocks or log segments found. Creating new log. I0507 06:32:59.111721 13716 log.cc:525] T d3b2287a051d4a8881adb4756316aaa1 P 89dd992cec6b407ca4e21db5e6aeef70: Log is configured to *not* fsync() on all Append() calls I0507 06:32:59.112150 13714 tablet_bootstrap.cc:590] T d3b2287a051d4a8881adb4756316aaa1 P b8748b85a6fb4418a992adbdfb1a0a62: No blocks or log segments found. Creating new log. I0507 06:32:59.113437 13714 log.cc:525] T d3b2287a051d4a8881adb4756316aaa1 P b8748b85a6fb4418a992adbdfb1a0a62: Log is configured to *not* fsync() on all Append() calls I0507 06:32:59.114665 13716 tablet_bootstrap.cc:439] T d3b2287a051d4a8881adb4756316aaa1 P 89dd992cec6b407ca4e21db5e6aeef70: No bootstrap required, opened a new log I0507 06:32:59.114825 13715 tablet_bootstrap.cc:439] T d3b2287a051d4a8881adb4756316aaa1 P 86eeb3337bd640599d56daee2302ae67: No bootstrap required, opened a new log I0507 06:32:59.114984 13716 ts_tablet_manager.cc:1053] T d3b2287a051d4a8881adb4756316aaa1 P 89dd992cec6b407ca4e21db5e6aeef70: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s I0507 06:32:59.115177 13715 ts_tablet_manager.cc:1053] T d3b2287a051d4a8881adb4756316aaa1 P 86eeb3337bd640599d56daee2302ae67: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.000s I0507 06:32:59.115201 13714 tablet_bootstrap.cc:439] T d3b2287a051d4a8881adb4756316aaa1 P b8748b85a6fb4418a992adbdfb1a0a62: No bootstrap required, opened a new log I0507 06:32:59.115465 13714 ts_tablet_manager.cc:1053] T d3b2287a051d4a8881adb4756316aaa1 P b8748b85a6fb4418a992adbdfb1a0a62: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.003s I0507 06:32:59.121695 13715 raft_consensus.cc:340] T d3b2287a051d4a8881adb4756316aaa1 P 86eeb3337bd640599d56daee2302ae67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8748b85a6fb4418a992adbdfb1a0a62" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 37113 } } peers { permanent_uuid: "89dd992cec6b407ca4e21db5e6aeef70" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 40533 } } peers { permanent_uuid: "86eeb3337bd640599d56daee2302ae67" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 37763 } } I0507 06:32:59.122089 13715 raft_consensus.cc:366] T d3b2287a051d4a8881adb4756316aaa1 P 86eeb3337bd640599d56daee2302ae67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0507 06:32:59.122144 13715 raft_consensus.cc:705] T d3b2287a051d4a8881adb4756316aaa1 P 86eeb3337bd640599d56daee2302ae67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86eeb3337bd640599d56daee2302ae67, State: Initialized, Role: FOLLOWER I0507 06:32:59.122674 13715 consensus_queue.cc:229] T d3b2287a051d4a8881adb4756316aaa1 P 86eeb3337bd640599d56daee2302ae67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8748b85a6fb4418a992adbdfb1a0a62" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 37113 } } peers { permanent_uuid: "89dd992cec6b407ca4e21db5e6aeef70" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 40533 } } peers { permanent_uuid: "86eeb3337bd640599d56daee2302ae67" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 37763 } } I0507 06:32:59.123500 13716 raft_consensus.cc:340] T d3b2287a051d4a8881adb4756316aaa1 P 89dd992cec6b407ca4e21db5e6aeef70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8748b85a6fb4418a992adbdfb1a0a62" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 37113 } } peers { permanent_uuid: "89dd992cec6b407ca4e21db5e6aeef70" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 40533 } } peers { permanent_uuid: "86eeb3337bd640599d56daee2302ae67" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 37763 } } I0507 06:32:59.123500 13714 raft_consensus.cc:340] T d3b2287a051d4a8881adb4756316aaa1 P b8748b85a6fb4418a992adbdfb1a0a62 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8748b85a6fb4418a992adbdfb1a0a62" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 37113 } } peers { permanent_uuid: "89dd992cec6b407ca4e21db5e6aeef70" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 40533 } } peers { permanent_uuid: "86eeb3337bd640599d56daee2302ae67" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 37763 } } I0507 06:32:59.123939 13714 raft_consensus.cc:366] T d3b2287a051d4a8881adb4756316aaa1 P b8748b85a6fb4418a992adbdfb1a0a62 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0507 06:32:59.124032 13716 raft_consensus.cc:366] T d3b2287a051d4a8881adb4756316aaa1 P 89dd992cec6b407ca4e21db5e6aeef70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0507 06:32:59.124146 13716 raft_consensus.cc:705] T d3b2287a051d4a8881adb4756316aaa1 P 89dd992cec6b407ca4e21db5e6aeef70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89dd992cec6b407ca4e21db5e6aeef70, State: Initialized, Role: FOLLOWER I0507 06:32:59.124455 13714 raft_consensus.cc:705] T d3b2287a051d4a8881adb4756316aaa1 P b8748b85a6fb4418a992adbdfb1a0a62 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8748b85a6fb4418a992adbdfb1a0a62, State: Initialized, Role: FOLLOWER I0507 06:32:59.124497 13573 heartbeater.cc:467] Master 127.0.57.254:37965 was elected leader, sending a full tablet report... I0507 06:32:59.124840 13716 consensus_queue.cc:229] T d3b2287a051d4a8881adb4756316aaa1 P 89dd992cec6b407ca4e21db5e6aeef70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8748b85a6fb4418a992adbdfb1a0a62" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 37113 } } peers { permanent_uuid: "89dd992cec6b407ca4e21db5e6aeef70" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 40533 } } peers { permanent_uuid: "86eeb3337bd640599d56daee2302ae67" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 37763 } } I0507 06:32:59.124975 13714 consensus_queue.cc:229] T d3b2287a051d4a8881adb4756316aaa1 P b8748b85a6fb4418a992adbdfb1a0a62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8748b85a6fb4418a992adbdfb1a0a62" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 37113 } } peers { permanent_uuid: "89dd992cec6b407ca4e21db5e6aeef70" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 40533 } } peers { permanent_uuid: "86eeb3337bd640599d56daee2302ae67" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 37763 } } I0507 06:32:59.125816 13450 heartbeater.cc:467] Master 127.0.57.254:37965 was elected leader, sending a full tablet report... I0507 06:32:59.126050 13715 ts_tablet_manager.cc:1081] T d3b2287a051d4a8881adb4756316aaa1 P 86eeb3337bd640599d56daee2302ae67: Time spent starting tablet: real 0.011s user 0.010s sys 0.000s I0507 06:32:59.126163 13695 heartbeater.cc:467] Master 127.0.57.254:37965 was elected leader, sending a full tablet report... I0507 06:32:59.127218 13716 ts_tablet_manager.cc:1081] T d3b2287a051d4a8881adb4756316aaa1 P 89dd992cec6b407ca4e21db5e6aeef70: Time spent starting tablet: real 0.012s user 0.006s sys 0.004s I0507 06:32:59.127957 13714 ts_tablet_manager.cc:1081] T d3b2287a051d4a8881adb4756316aaa1 P b8748b85a6fb4418a992adbdfb1a0a62: Time spent starting tablet: real 0.012s user 0.005s sys 0.005s W0507 06:32:59.130223 13574 tablet.cc:1786] T d3b2287a051d4a8881adb4756316aaa1 P 86eeb3337bd640599d56daee2302ae67: Can't schedule compaction. Clean time has not been advanced past its initial value. W0507 06:32:59.210461 13451 tablet.cc:1786] T d3b2287a051d4a8881adb4756316aaa1 P 89dd992cec6b407ca4e21db5e6aeef70: Can't schedule compaction. Clean time has not been advanced past its initial value. W0507 06:32:59.211755 13696 tablet.cc:1786] T d3b2287a051d4a8881adb4756316aaa1 P b8748b85a6fb4418a992adbdfb1a0a62: Can't schedule compaction. Clean time has not been advanced past its initial value. I0507 06:32:59.249435 13721 raft_consensus.cc:472] T d3b2287a051d4a8881adb4756316aaa1 P 89dd992cec6b407ca4e21db5e6aeef70 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0507 06:32:59.249732 13721 raft_consensus.cc:494] T d3b2287a051d4a8881adb4756316aaa1 P 89dd992cec6b407ca4e21db5e6aeef70 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8748b85a6fb4418a992adbdfb1a0a62" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 37113 } } peers { permanent_uuid: "89dd992cec6b407ca4e21db5e6aeef70" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 40533 } } peers { permanent_uuid: "86eeb3337bd640599d56daee2302ae67" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 37763 } } I0507 06:32:59.251312 13721 leader_election.cc:296] T d3b2287a051d4a8881adb4756316aaa1 P 89dd992cec6b407ca4e21db5e6aeef70 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b8748b85a6fb4418a992adbdfb1a0a62 (127.0.57.195:37113), 86eeb3337bd640599d56daee2302ae67 (127.0.57.194:37763) I0507 06:32:59.259043 13530 tablet_service.cc:1286] Received RequestConsensusVote() RPC: tablet_id: "d3b2287a051d4a8881adb4756316aaa1" candidate_uuid: "89dd992cec6b407ca4e21db5e6aeef70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86eeb3337bd640599d56daee2302ae67" is_pre_election: true I0507 06:32:59.259017 13652 tablet_service.cc:1286] Received RequestConsensusVote() RPC: tablet_id: "d3b2287a051d4a8881adb4756316aaa1" candidate_uuid: "89dd992cec6b407ca4e21db5e6aeef70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8748b85a6fb4418a992adbdfb1a0a62" is_pre_election: true I0507 06:32:59.259440 13530 raft_consensus.cc:2346] T d3b2287a051d4a8881adb4756316aaa1 P 86eeb3337bd640599d56daee2302ae67 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89dd992cec6b407ca4e21db5e6aeef70 in term 0. I0507 06:32:59.259441 13652 raft_consensus.cc:2346] T d3b2287a051d4a8881adb4756316aaa1 P b8748b85a6fb4418a992adbdfb1a0a62 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89dd992cec6b407ca4e21db5e6aeef70 in term 0. I0507 06:32:59.260176 13341 leader_election.cc:310] T d3b2287a051d4a8881adb4756316aaa1 P 89dd992cec6b407ca4e21db5e6aeef70 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 86eeb3337bd640599d56daee2302ae67, 89dd992cec6b407ca4e21db5e6aeef70; no voters: I0507 06:32:59.260749 13721 raft_consensus.cc:2640] T d3b2287a051d4a8881adb4756316aaa1 P 89dd992cec6b407ca4e21db5e6aeef70 [term 0 FOLLOWER]: Leader pre-election won for term 1 I0507 06:32:59.260852 13721 raft_consensus.cc:472] T d3b2287a051d4a8881adb4756316aaa1 P 89dd992cec6b407ca4e21db5e6aeef70 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0507 06:32:59.260908 13721 raft_consensus.cc:2886] T d3b2287a051d4a8881adb4756316aaa1 P 89dd992cec6b407ca4e21db5e6aeef70 [term 0 FOLLOWER]: Advancing to term 1 I0507 06:32:59.262917 13721 raft_consensus.cc:494] T d3b2287a051d4a8881adb4756316aaa1 P 89dd992cec6b407ca4e21db5e6aeef70 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8748b85a6fb4418a992adbdfb1a0a62" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 37113 } } peers { permanent_uuid: "89dd992cec6b407ca4e21db5e6aeef70" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 40533 } } peers { permanent_uuid: "86eeb3337bd640599d56daee2302ae67" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 37763 } } I0507 06:32:59.263471 13721 leader_election.cc:296] T d3b2287a051d4a8881adb4756316aaa1 P 89dd992cec6b407ca4e21db5e6aeef70 [CANDIDATE]: Term 1 election: Requested vote from peers b8748b85a6fb4418a992adbdfb1a0a62 (127.0.57.195:37113), 86eeb3337bd640599d56daee2302ae67 (127.0.57.194:37763) I0507 06:32:59.264053 13652 tablet_service.cc:1286] Received RequestConsensusVote() RPC: tablet_id: "d3b2287a051d4a8881adb4756316aaa1" candidate_uuid: "89dd992cec6b407ca4e21db5e6aeef70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8748b85a6fb4418a992adbdfb1a0a62" I0507 06:32:59.264041 13530 tablet_service.cc:1286] Received RequestConsensusVote() RPC: tablet_id: "d3b2287a051d4a8881adb4756316aaa1" candidate_uuid: "89dd992cec6b407ca4e21db5e6aeef70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86eeb3337bd640599d56daee2302ae67" I0507 06:32:59.264227 13652 raft_consensus.cc:2886] T d3b2287a051d4a8881adb4756316aaa1 P b8748b85a6fb4418a992adbdfb1a0a62 [term 0 FOLLOWER]: Advancing to term 1 I0507 06:32:59.264241 13530 raft_consensus.cc:2886] T d3b2287a051d4a8881adb4756316aaa1 P 86eeb3337bd640599d56daee2302ae67 [term 0 FOLLOWER]: Advancing to term 1 I0507 06:32:59.266633 13530 raft_consensus.cc:2346] T d3b2287a051d4a8881adb4756316aaa1 P 86eeb3337bd640599d56daee2302ae67 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89dd992cec6b407ca4e21db5e6aeef70 in term 1. I0507 06:32:59.266870 13652 raft_consensus.cc:2346] T d3b2287a051d4a8881adb4756316aaa1 P b8748b85a6fb4418a992adbdfb1a0a62 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89dd992cec6b407ca4e21db5e6aeef70 in term 1. I0507 06:32:59.267050 13341 leader_election.cc:310] T d3b2287a051d4a8881adb4756316aaa1 P 89dd992cec6b407ca4e21db5e6aeef70 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 86eeb3337bd640599d56daee2302ae67, 89dd992cec6b407ca4e21db5e6aeef70; no voters: I0507 06:32:59.267391 13721 raft_consensus.cc:2640] T d3b2287a051d4a8881adb4756316aaa1 P 89dd992cec6b407ca4e21db5e6aeef70 [term 1 FOLLOWER]: Leader election won for term 1 I0507 06:32:59.267956 13721 raft_consensus.cc:667] T d3b2287a051d4a8881adb4756316aaa1 P 89dd992cec6b407ca4e21db5e6aeef70 [term 1 LEADER]: Becoming Leader. State: Replica: 89dd992cec6b407ca4e21db5e6aeef70, State: Running, Role: LEADER I0507 06:32:59.268404 13721 consensus_queue.cc:206] T d3b2287a051d4a8881adb4756316aaa1 P 89dd992cec6b407ca4e21db5e6aeef70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8748b85a6fb4418a992adbdfb1a0a62" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 37113 } } peers { permanent_uuid: "89dd992cec6b407ca4e21db5e6aeef70" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 40533 } } peers { permanent_uuid: "86eeb3337bd640599d56daee2302ae67" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 37763 } } I0507 06:32:59.272727 13285 catalog_manager.cc:3988] T d3b2287a051d4a8881adb4756316aaa1 P 89dd992cec6b407ca4e21db5e6aeef70 reported cstate change: term changed from 0 to 1, leader changed from to 89dd992cec6b407ca4e21db5e6aeef70 (127.0.57.193). New cstate: current_term: 1 leader_uuid: "89dd992cec6b407ca4e21db5e6aeef70" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8748b85a6fb4418a992adbdfb1a0a62" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 37113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "89dd992cec6b407ca4e21db5e6aeef70" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 40533 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "86eeb3337bd640599d56daee2302ae67" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 37763 } health_report { overall_health: UNKNOWN } } } I0507 06:32:59.317895 231 external_mini_cluster.cc:623] 3 TS(s) registered with all masters I0507 06:32:59.319540 231 ts_itest-base.cc:232] Waiting for 1 tablets on tserver 89dd992cec6b407ca4e21db5e6aeef70 to finish bootstrapping I0507 06:32:59.326328 231 ts_itest-base.cc:232] Waiting for 1 tablets on tserver 86eeb3337bd640599d56daee2302ae67 to finish bootstrapping I0507 06:32:59.332906 231 ts_itest-base.cc:232] Waiting for 1 tablets on tserver b8748b85a6fb4418a992adbdfb1a0a62 to finish bootstrapping I0507 06:32:59.353718 231 cluster_itest_util.cc:258] Not converged past 1 yet: 1.1 0.0 0.0 I0507 06:32:59.458127 231 cluster_itest_util.cc:258] Not converged past 1 yet: 1.1 0.0 0.0 I0507 06:32:59.649596 13652 raft_consensus.cc:1181] T d3b2287a051d4a8881adb4756316aaa1 P b8748b85a6fb4418a992adbdfb1a0a62 [term 1 FOLLOWER]: Refusing update from remote peer 89dd992cec6b407ca4e21db5e6aeef70: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch) I0507 06:32:59.650511 13721 consensus_queue.cc:984] T d3b2287a051d4a8881adb4756316aaa1 P 89dd992cec6b407ca4e21db5e6aeef70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8748b85a6fb4418a992adbdfb1a0a62" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 37113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0507 06:32:59.661350 13530 raft_consensus.cc:1181] T d3b2287a051d4a8881adb4756316aaa1 P 86eeb3337bd640599d56daee2302ae67 [term 1 FOLLOWER]: Refusing update from remote peer 89dd992cec6b407ca4e21db5e6aeef70: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch) I0507 06:32:59.662380 13721 consensus_queue.cc:984] T d3b2287a051d4a8881adb4756316aaa1 P 89dd992cec6b407ca4e21db5e6aeef70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "86eeb3337bd640599d56daee2302ae67" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 37763 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0507 06:32:59.663471 231 cluster_itest_util.cc:258] Not converged past 1 yet: 1.1 0.0 1.1 I0507 06:33:14.983067 231 test_util.cc:212] Using random seed: 1682341458 I0507 06:33:14.998615 231 test_workload.cc:306] TestWorkload: Skipping table creation because table TestTable already exists I0507 06:33:15.000681 231 raft_consensus-itest-base.cc:271] Waiting until we've written at least 4MB... I0507 06:33:15.142877 231 raft_consensus-itest-base.cc:277] Waiting for log GC on 89dd992cec6b407ca4e21db5e6aeef70 I0507 06:33:15.164292 13451 maintenance_manager.cc:306] P 89dd992cec6b407ca4e21db5e6aeef70: Scheduling FlushMRSOp(d3b2287a051d4a8881adb4756316aaa1): 4602000 bytes log retention I0507 06:33:15.164320 13696 maintenance_manager.cc:306] P b8748b85a6fb4418a992adbdfb1a0a62: Scheduling FlushMRSOp(d3b2287a051d4a8881adb4756316aaa1): 4338862 bytes log retention I0507 06:33:15.232973 13590 maintenance_manager.cc:501] P b8748b85a6fb4418a992adbdfb1a0a62: FlushMRSOp(d3b2287a051d4a8881adb4756316aaa1) complete. Timing: real 0.066s user 0.058s sys 0.007s Metrics: {"bytes_written":136001,"cfile_init":1,"compiler_manager_pool.queue_time_us":433,"data dirs.queue_time_us":1835,"data dirs.run_cpu_time_us":241,"data dirs.run_wall_time_us":3475,"drs_written":1,"lbm_read_time_us":68,"lbm_reads_lt_1ms":4,"lbm_write_time_us":526,"lbm_writes_lt_1ms":23,"rows_written":37,"thread_start_us":1760,"threads_started":2} I0507 06:33:15.234426 13696 maintenance_manager.cc:306] P b8748b85a6fb4418a992adbdfb1a0a62: Scheduling LogGCOp(d3b2287a051d4a8881adb4756316aaa1): free 2629629 bytes of WAL I0507 06:33:15.234899 13590 log_reader.cc:377] T d3b2287a051d4a8881adb4756316aaa1: removed 3 log segments from log reader I0507 06:33:15.235054 13590 log.cc:863] T d3b2287a051d4a8881adb4756316aaa1 P b8748b85a6fb4418a992adbdfb1a0a62: Deleting log segment in path: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_1.1557210500371863-231/raft_consensus-itest-cluster/ts-2/wal/wals/d3b2287a051d4a8881adb4756316aaa1/wal-000000001 (ops 1-7) I0507 06:33:15.235183 13590 log.cc:863] T d3b2287a051d4a8881adb4756316aaa1 P b8748b85a6fb4418a992adbdfb1a0a62: Deleting log segment in path: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_1.1557210500371863-231/raft_consensus-itest-cluster/ts-2/wal/wals/d3b2287a051d4a8881adb4756316aaa1/wal-000000002 (ops 8-14) I0507 06:33:15.235280 13590 log.cc:863] T d3b2287a051d4a8881adb4756316aaa1 P b8748b85a6fb4418a992adbdfb1a0a62: Deleting log segment in path: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_1.1557210500371863-231/raft_consensus-itest-cluster/ts-2/wal/wals/d3b2287a051d4a8881adb4756316aaa1/wal-000000003 (ops 15-21) I0507 06:33:15.235929 13590 maintenance_manager.cc:501] P b8748b85a6fb4418a992adbdfb1a0a62: LogGCOp(d3b2287a051d4a8881adb4756316aaa1) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {} I0507 06:33:15.236243 13696 maintenance_manager.cc:306] P b8748b85a6fb4418a992adbdfb1a0a62: Scheduling UndoDeltaBlockGCOp(d3b2287a051d4a8881adb4756316aaa1): 550 bytes on disk I0507 06:33:15.237009 13590 maintenance_manager.cc:501] P b8748b85a6fb4418a992adbdfb1a0a62: UndoDeltaBlockGCOp(d3b2287a051d4a8881adb4756316aaa1) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":74,"lbm_reads_lt_1ms":4} I0507 06:33:15.296597 13345 maintenance_manager.cc:501] P 89dd992cec6b407ca4e21db5e6aeef70: FlushMRSOp(d3b2287a051d4a8881adb4756316aaa1) complete. Timing: real 0.132s user 0.108s sys 0.013s Metrics: {"bytes_written":136001,"cfile_init":1,"compiler_manager_pool.queue_time_us":12149,"data dirs.queue_time_us":2729,"data dirs.run_cpu_time_us":217,"data dirs.run_wall_time_us":875,"drs_written":1,"lbm_read_time_us":119,"lbm_reads_lt_1ms":4,"lbm_write_time_us":798,"lbm_writes_lt_1ms":23,"rows_written":37,"thread_start_us":16775,"threads_started":2} I0507 06:33:15.297960 13451 maintenance_manager.cc:306] P 89dd992cec6b407ca4e21db5e6aeef70: Scheduling LogGCOp(d3b2287a051d4a8881adb4756316aaa1): free 2761146 bytes of WAL I0507 06:33:15.298321 13345 log_reader.cc:377] T d3b2287a051d4a8881adb4756316aaa1: removed 3 log segments from log reader I0507 06:33:15.298415 13345 log.cc:863] T d3b2287a051d4a8881adb4756316aaa1 P 89dd992cec6b407ca4e21db5e6aeef70: Deleting log segment in path: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_1.1557210500371863-231/raft_consensus-itest-cluster/ts-0/wal/wals/d3b2287a051d4a8881adb4756316aaa1/wal-000000001 (ops 1-8) I0507 06:33:15.298502 13345 log.cc:863] T d3b2287a051d4a8881adb4756316aaa1 P 89dd992cec6b407ca4e21db5e6aeef70: Deleting log segment in path: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_1.1557210500371863-231/raft_consensus-itest-cluster/ts-0/wal/wals/d3b2287a051d4a8881adb4756316aaa1/wal-000000002 (ops 9-15) I0507 06:33:15.298583 13345 log.cc:863] T d3b2287a051d4a8881adb4756316aaa1 P 89dd992cec6b407ca4e21db5e6aeef70: Deleting log segment in path: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_1.1557210500371863-231/raft_consensus-itest-cluster/ts-0/wal/wals/d3b2287a051d4a8881adb4756316aaa1/wal-000000003 (ops 16-22) I0507 06:33:15.299495 13345 maintenance_manager.cc:501] P 89dd992cec6b407ca4e21db5e6aeef70: LogGCOp(d3b2287a051d4a8881adb4756316aaa1) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {} I0507 06:33:15.299830 13451 maintenance_manager.cc:306] P 89dd992cec6b407ca4e21db5e6aeef70: Scheduling UndoDeltaBlockGCOp(d3b2287a051d4a8881adb4756316aaa1): 550 bytes on disk I0507 06:33:15.300812 13345 maintenance_manager.cc:501] P 89dd992cec6b407ca4e21db5e6aeef70: UndoDeltaBlockGCOp(d3b2287a051d4a8881adb4756316aaa1) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":98,"lbm_reads_lt_1ms":4} I0507 06:33:15.307112 231 raft_consensus-itest-base.cc:282] Log GC complete on 89dd992cec6b407ca4e21db5e6aeef70 I0507 06:33:17.309401 231 raft_consensus-itest-base.cc:298] Servers converged with original term 1 I0507 06:33:17.309469 231 raft_consensus-itest-base.cc:303] Resuming 86eeb3337bd640599d56daee2302ae67 I0507 06:33:17.310988 13733 raft_consensus.cc:472] T d3b2287a051d4a8881adb4756316aaa1 P 86eeb3337bd640599d56daee2302ae67 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 89dd992cec6b407ca4e21db5e6aeef70) I0507 06:33:17.311164 13733 raft_consensus.cc:494] T d3b2287a051d4a8881adb4756316aaa1 P 86eeb3337bd640599d56daee2302ae67 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8748b85a6fb4418a992adbdfb1a0a62" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 37113 } } peers { permanent_uuid: "89dd992cec6b407ca4e21db5e6aeef70" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 40533 } } peers { permanent_uuid: "86eeb3337bd640599d56daee2302ae67" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 37763 } } I0507 06:33:17.313207 13733 leader_election.cc:296] T d3b2287a051d4a8881adb4756316aaa1 P 86eeb3337bd640599d56daee2302ae67 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b8748b85a6fb4418a992adbdfb1a0a62 (127.0.57.195:37113), 89dd992cec6b407ca4e21db5e6aeef70 (127.0.57.193:40533) I0507 06:33:17.313277 13782 consensus_queue.cc:689] T d3b2287a051d4a8881adb4756316aaa1 P 89dd992cec6b407ca4e21db5e6aeef70 [LEADER]: The logs necessary to catch up peer 86eeb3337bd640599d56daee2302ae67 have been garbage collected. The follower will never be able to catch up (Not found: Failed to read ops 2..31: Segment 1 which contained index 2 has been GCed) I0507 06:33:17.313522 13782 consensus_queue.cc:544] T d3b2287a051d4a8881adb4756316aaa1 P 89dd992cec6b407ca4e21db5e6aeef70 [LEADER]: The logs necessary to catch up peer 86eeb3337bd640599d56daee2302ae67 have been garbage collected. The replica will never be able to catch up I0507 06:33:17.318403 13284 catalog_manager.cc:3739] Sending ChangeConfig:REMOVE_PEER on tablet d3b2287a051d4a8881adb4756316aaa1 (attempt 1) I0507 06:33:17.320269 13407 consensus_queue.cc:206] T d3b2287a051d4a8881adb4756316aaa1 P 89dd992cec6b407ca4e21db5e6aeef70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 38, Committed index: 38, Last appended: 1.38, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 39 OBSOLETE_local: false peers { permanent_uuid: "b8748b85a6fb4418a992adbdfb1a0a62" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 37113 } } peers { permanent_uuid: "89dd992cec6b407ca4e21db5e6aeef70" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 40533 } } I0507 06:33:17.322381 13652 raft_consensus.cc:1181] T d3b2287a051d4a8881adb4756316aaa1 P b8748b85a6fb4418a992adbdfb1a0a62 [term 1 FOLLOWER]: Refusing update from remote peer 89dd992cec6b407ca4e21db5e6aeef70: Log matching property violated. Preceding OpId in replica: term: 1 index: 38. Preceding OpId from leader: term: 1 index: 39. (index mismatch) I0507 06:33:17.323050 13786 consensus_queue.cc:984] T d3b2287a051d4a8881adb4756316aaa1 P 89dd992cec6b407ca4e21db5e6aeef70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8748b85a6fb4418a992adbdfb1a0a62" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 37113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 39, Last known committed idx: 38, Time since last communication: 0.000s I0507 06:33:17.326279 13783 raft_consensus.cc:2788] T d3b2287a051d4a8881adb4756316aaa1 P 89dd992cec6b407ca4e21db5e6aeef70 [term 1 LEADER]: Committing config change with OpId 1.39: config changed from index -1 to 39, VOTER 86eeb3337bd640599d56daee2302ae67 (127.0.57.194) evicted. New config: { opid_index: 39 OBSOLETE_local: false peers { permanent_uuid: "b8748b85a6fb4418a992adbdfb1a0a62" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 37113 } } peers { permanent_uuid: "89dd992cec6b407ca4e21db5e6aeef70" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 40533 } } } I0507 06:33:17.326987 13652 raft_consensus.cc:2788] T d3b2287a051d4a8881adb4756316aaa1 P b8748b85a6fb4418a992adbdfb1a0a62 [term 1 FOLLOWER]: Committing config change with OpId 1.39: config changed from index -1 to 39, VOTER 86eeb3337bd640599d56daee2302ae67 (127.0.57.194) evicted. New config: { opid_index: 39 OBSOLETE_local: false peers { permanent_uuid: "b8748b85a6fb4418a992adbdfb1a0a62" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 37113 } } peers { permanent_uuid: "89dd992cec6b407ca4e21db5e6aeef70" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 40533 } } } I0507 06:33:17.329231 13407 tablet_service.cc:1286] Received RequestConsensusVote() RPC: tablet_id: "d3b2287a051d4a8881adb4756316aaa1" candidate_uuid: "86eeb3337bd640599d56daee2302ae67" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "89dd992cec6b407ca4e21db5e6aeef70" is_pre_election: true I0507 06:33:17.330296 13268 catalog_manager.cc:3531] ChangeConfig:REMOVE_PEER RPC for tablet d3b2287a051d4a8881adb4756316aaa1 with cas_config_opid_index -1: ChangeConfig:REMOVE_PEER succeeded (attempt 1) I0507 06:33:17.330632 13407 raft_consensus.cc:1704] T d3b2287a051d4a8881adb4756316aaa1 P 89dd992cec6b407ca4e21db5e6aeef70 [term 1 LEADER]: Handling vote request from an unknown peer 86eeb3337bd640599d56daee2302ae67 I0507 06:33:17.330736 13407 raft_consensus.cc:2307] T d3b2287a051d4a8881adb4756316aaa1 P 89dd992cec6b407ca4e21db5e6aeef70 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate 86eeb3337bd640599d56daee2302ae67 for term 2 because replica is either leader or believes a valid leader to be alive. I0507 06:33:17.331665 13652 tablet_service.cc:1286] Received RequestConsensusVote() RPC: tablet_id: "d3b2287a051d4a8881adb4756316aaa1" candidate_uuid: "86eeb3337bd640599d56daee2302ae67" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b8748b85a6fb4418a992adbdfb1a0a62" is_pre_election: true I0507 06:33:17.331528 13285 catalog_manager.cc:3988] T d3b2287a051d4a8881adb4756316aaa1 P b8748b85a6fb4418a992adbdfb1a0a62 reported cstate change: config changed from index -1 to 39, VOTER 86eeb3337bd640599d56daee2302ae67 (127.0.57.194) evicted. New cstate: current_term: 1 leader_uuid: "89dd992cec6b407ca4e21db5e6aeef70" committed_config { opid_index: 39 OBSOLETE_local: false peers { permanent_uuid: "b8748b85a6fb4418a992adbdfb1a0a62" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 37113 } } peers { permanent_uuid: "89dd992cec6b407ca4e21db5e6aeef70" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 40533 } } } I0507 06:33:17.331794 13652 raft_consensus.cc:1704] T d3b2287a051d4a8881adb4756316aaa1 P b8748b85a6fb4418a992adbdfb1a0a62 [term 1 FOLLOWER]: Handling vote request from an unknown peer 86eeb3337bd640599d56daee2302ae67 I0507 06:33:17.331897 13652 raft_consensus.cc:2307] T d3b2287a051d4a8881adb4756316aaa1 P b8748b85a6fb4418a992adbdfb1a0a62 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 86eeb3337bd640599d56daee2302ae67 for term 2 because replica is either leader or believes a valid leader to be alive. I0507 06:33:17.332265 13462 leader_election.cc:310] T d3b2287a051d4a8881adb4756316aaa1 P 86eeb3337bd640599d56daee2302ae67 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 86eeb3337bd640599d56daee2302ae67; no voters: 89dd992cec6b407ca4e21db5e6aeef70, b8748b85a6fb4418a992adbdfb1a0a62 I0507 06:33:17.332760 13733 raft_consensus.cc:2588] T d3b2287a051d4a8881adb4756316aaa1 P 86eeb3337bd640599d56daee2302ae67 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority I0507 06:33:17.340324 13285 catalog_manager.cc:3390] Sending DeleteTablet(TABLET_DATA_TOMBSTONED) for tablet d3b2287a051d4a8881adb4756316aaa1 on 86eeb3337bd640599d56daee2302ae67 (127.0.57.194:37763) (TS 86eeb3337bd640599d56daee2302ae67 not found in new config with opid_index 39) I0507 06:33:17.341303 13510 tablet_service.cc:1057] Processing DeleteTablet for tablet d3b2287a051d4a8881adb4756316aaa1 with delete_type TABLET_DATA_TOMBSTONED (TS 86eeb3337bd640599d56daee2302ae67 not found in new config with opid_index 39) from {username='slave'} at 127.0.0.1:39764 I0507 06:33:17.342121 13796 tablet_replica.cc:264] T d3b2287a051d4a8881adb4756316aaa1 P 86eeb3337bd640599d56daee2302ae67: stopping tablet replica I0507 06:33:17.342404 13796 raft_consensus.cc:2143] T d3b2287a051d4a8881adb4756316aaa1 P 86eeb3337bd640599d56daee2302ae67 [term 1 FOLLOWER]: Raft consensus shutting down. I0507 06:33:17.342691 13796 raft_consensus.cc:2170] T d3b2287a051d4a8881adb4756316aaa1 P 86eeb3337bd640599d56daee2302ae67 [term 1 FOLLOWER]: Raft consensus is shut down! I0507 06:33:17.343924 13796 ts_tablet_manager.cc:1377] T d3b2287a051d4a8881adb4756316aaa1 P 86eeb3337bd640599d56daee2302ae67: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED I0507 06:33:17.345289 13696 maintenance_manager.cc:306] P b8748b85a6fb4418a992adbdfb1a0a62: Scheduling LogGCOp(d3b2287a051d4a8881adb4756316aaa1): free 1709233 bytes of WAL I0507 06:33:17.345660 13590 log_reader.cc:377] T d3b2287a051d4a8881adb4756316aaa1: removed 2 log segments from log reader I0507 06:33:17.345732 13590 log.cc:863] T d3b2287a051d4a8881adb4756316aaa1 P b8748b85a6fb4418a992adbdfb1a0a62: Deleting log segment in path: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_1.1557210500371863-231/raft_consensus-itest-cluster/ts-2/wal/wals/d3b2287a051d4a8881adb4756316aaa1/wal-000000004 (ops 22-27) I0507 06:33:17.345811 13590 log.cc:863] T d3b2287a051d4a8881adb4756316aaa1 P b8748b85a6fb4418a992adbdfb1a0a62: Deleting log segment in path: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_1.1557210500371863-231/raft_consensus-itest-cluster/ts-2/wal/wals/d3b2287a051d4a8881adb4756316aaa1/wal-000000005 (ops 28-34) I0507 06:33:17.346249 13590 maintenance_manager.cc:501] P b8748b85a6fb4418a992adbdfb1a0a62: LogGCOp(d3b2287a051d4a8881adb4756316aaa1) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {} I0507 06:33:17.347486 13796 ts_tablet_manager.cc:1390] T d3b2287a051d4a8881adb4756316aaa1 P 86eeb3337bd640599d56daee2302ae67: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.1 I0507 06:33:17.347580 13796 log.cc:990] T d3b2287a051d4a8881adb4756316aaa1 P 86eeb3337bd640599d56daee2302ae67: Deleting WAL directory at /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_1.1557210500371863-231/raft_consensus-itest-cluster/ts-1/wal/wals/d3b2287a051d4a8881adb4756316aaa1 I0507 06:33:17.348548 13270 catalog_manager.cc:3368] TS 86eeb3337bd640599d56daee2302ae67 (127.0.57.194:37763): tablet d3b2287a051d4a8881adb4756316aaa1 (table TestTable [id=26087773aad9435a9eef387b8101db47]) successfully deleted I0507 06:33:17.363353 231 external_mini_cluster.cc:1172] Attempting to check leaks for /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-master pid 13260 I0507 06:33:17.364068 13316 generic_service.cc:171] Checking for leaks (request via RPC) I0507 06:33:17.409770 13451 maintenance_manager.cc:306] P 89dd992cec6b407ca4e21db5e6aeef70: Scheduling LogGCOp(d3b2287a051d4a8881adb4756316aaa1): free 1840854 bytes of WAL I0507 06:33:17.410377 13345 log_reader.cc:377] T d3b2287a051d4a8881adb4756316aaa1: removed 2 log segments from log reader I0507 06:33:17.410475 13345 log.cc:863] T d3b2287a051d4a8881adb4756316aaa1 P 89dd992cec6b407ca4e21db5e6aeef70: Deleting log segment in path: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_1.1557210500371863-231/raft_consensus-itest-cluster/ts-0/wal/wals/d3b2287a051d4a8881adb4756316aaa1/wal-000000004 (ops 23-29) I0507 06:33:17.410565 13345 log.cc:863] T d3b2287a051d4a8881adb4756316aaa1 P 89dd992cec6b407ca4e21db5e6aeef70: Deleting log segment in path: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_1.1557210500371863-231/raft_consensus-itest-cluster/ts-0/wal/wals/d3b2287a051d4a8881adb4756316aaa1/wal-000000005 (ops 30-36) I0507 06:33:17.411047 13345 maintenance_manager.cc:501] P 89dd992cec6b407ca4e21db5e6aeef70: LogGCOp(d3b2287a051d4a8881adb4756316aaa1) complete. Timing: real 0.001s user 0.001s sys 0.000s Metrics: {} I0507 06:33:17.450587 231 external_mini_cluster.cc:1131] Killing /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-master with pid 13260 I0507 06:33:17.466048 231 external_mini_cluster.cc:933] Running /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-master /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-master --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_1.1557210500371863-231/raft_consensus-itest-cluster/master-0/wal --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_1.1557210500371863-231/raft_consensus-itest-cluster/master-0/data --block_manager=log --webserver_interface=localhost --ipki_ca_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.0.57.254:37965 --webserver_interface=127.0.57.254 --webserver_port=40901 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_1.1557210500371863-231/raft_consensus-itest-cluster/master-0/logs --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_1.1557210500371863-231/raft_consensus-itest-cluster/master-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=true --master_add_server_when_underreplicated=false --rpc_reuseport=true --master_add_server_when_underreplicated=true with env {} WARNING: Logging before InitGoogleLogging() is written to STDERR W0507 06:33:17.687794 13798 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0507 06:33:17.688072 13798 flags.cc:404] Enabled unsafe flag: --never_fsync=true W0507 06:33:17.690675 13798 flags.cc:404] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true W0507 06:33:17.690749 13798 flags.cc:404] Enabled experimental flag: --ipki_ca_key_size=1024 W0507 06:33:17.690804 13798 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W0507 06:33:17.690838 13798 flags.cc:404] Enabled experimental flag: --tsk_num_rsa_bits=512 W0507 06:33:17.690883 13798 flags.cc:404] Enabled experimental flag: --rpc_reuseport=true I0507 06:33:17.695580 13798 master_main.cc:73] Master server non-default flags: --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_1.1557210500371863-231/raft_consensus-itest-cluster/master-0/data --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_1.1557210500371863-231/raft_consensus-itest-cluster/master-0/wal --ipki_ca_key_size=1024 --ipki_server_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.0.57.254:37965 --rpc_reuseport=true --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_1.1557210500371863-231/raft_consensus-itest-cluster/master-0/data/info.pb --webserver_interface=127.0.57.254 --webserver_port=40901 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_1.1557210500371863-231/raft_consensus-itest-cluster/master-0/logs --logbuflevel=-1 --logtostderr=true Master server version: kudu 1.10.0-cdh6.x-SNAPSHOT revision 7d0ee26697926e87af9ef395008799577f6ab43f build type FASTDEBUG built by jenkins at 06 May 2019 22:44:21 PST on quasar-knmpfx-1.vpc.cloudera.com build id 322017 ASAN enabled I0507 06:33:17.703073 13798 system_ntp.cc:202] NTP initialized. Skew: 500ppm Current error: 173016us I0507 06:33:17.704470 13798 fs_manager.cc:263] Metadata directory not provided I0507 06:33:17.704576 13798 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0507 06:33:17.711169 13798 fs_manager.cc:399] Time spent opening directory manager: real 0.005s user 0.003s sys 0.002s I0507 06:33:17.711977 13798 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0507 06:33:17.712707 13798 file_cache.cc:466] Constructed file cache lbm with capacity 419430 I0507 06:33:17.716063 13798 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s I0507 06:33:17.716172 13798 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_1.1557210500371863-231/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_1.1557210500371863-231/raft_consensus-itest-cluster/master-0/wal uuid: "0c8516a734564649b36e53a7f5184239" format_stamp: "Formatted at 2019-05-07 06:32:57 on dist-test-slave-dist-test-slave-rqp6" I0507 06:33:17.717264 13798 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_1.1557210500371863-231/raft_consensus-itest-cluster/master-0/wal metadata directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_1.1557210500371863-231/raft_consensus-itest-cluster/master-0/wal 1 data directories: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_1.1557210500371863-231/raft_consensus-itest-cluster/master-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0507 06:33:17.754381 13798 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0507 06:33:17.769987 13798 rpc_server.cc:205] RPC server started. Bound to: 127.0.57.254:37965 I0507 06:33:17.771384 13798 webserver.cc:308] Webserver started at http://127.0.57.254:40901/ using document root and password file I0507 06:33:17.772408 13798 server_base.cc:608] Dumped server information to /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_1.1557210500371863-231/raft_consensus-itest-cluster/master-0/data/info.pb I0507 06:33:17.773727 231 external_mini_cluster.cc:995] Started /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-master as pid 13798 I0507 06:33:17.783680 13857 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 0c8516a734564649b36e53a7f5184239: Bootstrap starting. I0507 06:33:17.799809 13857 log.cc:525] T 00000000000000000000000000000000 P 0c8516a734564649b36e53a7f5184239: Log is configured to *not* fsync() on all Append() calls I0507 06:33:17.809906 13857 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 0c8516a734564649b36e53a7f5184239: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=4 ignored=0} mutations{seen=3 ignored=0} orphaned_commits=0. Pending: 0 replicates I0507 06:33:17.810422 13857 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 0c8516a734564649b36e53a7f5184239: Bootstrap complete. I0507 06:33:17.817349 13857 raft_consensus.cc:340] T 00000000000000000000000000000000 P 0c8516a734564649b36e53a7f5184239 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c8516a734564649b36e53a7f5184239" member_type: VOTER } I0507 06:33:17.818120 13857 raft_consensus.cc:705] T 00000000000000000000000000000000 P 0c8516a734564649b36e53a7f5184239 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c8516a734564649b36e53a7f5184239, State: Initialized, Role: FOLLOWER I0507 06:33:17.818728 13857 consensus_queue.cc:229] T 00000000000000000000000000000000 P 0c8516a734564649b36e53a7f5184239 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c8516a734564649b36e53a7f5184239" member_type: VOTER } I0507 06:33:17.818913 13857 raft_consensus.cc:380] T 00000000000000000000000000000000 P 0c8516a734564649b36e53a7f5184239 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0507 06:33:17.818996 13857 raft_consensus.cc:472] T 00000000000000000000000000000000 P 0c8516a734564649b36e53a7f5184239 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0507 06:33:17.819108 13857 raft_consensus.cc:2886] T 00000000000000000000000000000000 P 0c8516a734564649b36e53a7f5184239 [term 1 FOLLOWER]: Advancing to term 2 I0507 06:33:17.823442 13857 raft_consensus.cc:494] T 00000000000000000000000000000000 P 0c8516a734564649b36e53a7f5184239 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c8516a734564649b36e53a7f5184239" member_type: VOTER } I0507 06:33:17.823942 13857 leader_election.cc:310] T 00000000000000000000000000000000 P 0c8516a734564649b36e53a7f5184239 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0c8516a734564649b36e53a7f5184239; no voters: I0507 06:33:17.824810 13857 leader_election.cc:296] T 00000000000000000000000000000000 P 0c8516a734564649b36e53a7f5184239 [CANDIDATE]: Term 2 election: Requested vote from peers I0507 06:33:17.825027 13861 raft_consensus.cc:2640] T 00000000000000000000000000000000 P 0c8516a734564649b36e53a7f5184239 [term 2 FOLLOWER]: Leader election won for term 2 I0507 06:33:17.825963 13861 raft_consensus.cc:667] T 00000000000000000000000000000000 P 0c8516a734564649b36e53a7f5184239 [term 2 LEADER]: Becoming Leader. State: Replica: 0c8516a734564649b36e53a7f5184239, State: Running, Role: LEADER I0507 06:33:17.826346 13861 consensus_queue.cc:206] T 00000000000000000000000000000000 P 0c8516a734564649b36e53a7f5184239 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c8516a734564649b36e53a7f5184239" member_type: VOTER } I0507 06:33:17.829030 13857 sys_catalog.cc:337] T 00000000000000000000000000000000 P 0c8516a734564649b36e53a7f5184239 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 2 leader_uuid: "0c8516a734564649b36e53a7f5184239" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c8516a734564649b36e53a7f5184239" member_type: VOTER } } I0507 06:33:17.829255 13857 sys_catalog.cc:340] T 00000000000000000000000000000000 P 0c8516a734564649b36e53a7f5184239 [sys.catalog]: This master's current role is: LEADER I0507 06:33:17.830485 13862 sys_catalog.cc:337] T 00000000000000000000000000000000 P 0c8516a734564649b36e53a7f5184239 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "0c8516a734564649b36e53a7f5184239" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c8516a734564649b36e53a7f5184239" member_type: VOTER } } I0507 06:33:17.830358 13863 sys_catalog.cc:337] T 00000000000000000000000000000000 P 0c8516a734564649b36e53a7f5184239 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0c8516a734564649b36e53a7f5184239. Latest consensus state: current_term: 2 leader_uuid: "0c8516a734564649b36e53a7f5184239" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c8516a734564649b36e53a7f5184239" member_type: VOTER } } I0507 06:33:17.831173 13862 sys_catalog.cc:340] T 00000000000000000000000000000000 P 0c8516a734564649b36e53a7f5184239 [sys.catalog]: This master's current role is: LEADER I0507 06:33:17.831202 13863 sys_catalog.cc:340] T 00000000000000000000000000000000 P 0c8516a734564649b36e53a7f5184239 [sys.catalog]: This master's current role is: LEADER I0507 06:33:17.831765 13857 sys_catalog.cc:424] T 00000000000000000000000000000000 P 0c8516a734564649b36e53a7f5184239 [sys.catalog]: configured and running, proceeding with master startup. I0507 06:33:17.832016 13867 catalog_manager.cc:1055] Loading table and tablet metadata into memory... I0507 06:33:17.836714 13867 catalog_manager.cc:356] Loaded metadata for table TestTable [id=26087773aad9435a9eef387b8101db47] I0507 06:33:17.838333 13867 catalog_manager.cc:410] Loaded metadata for tablet d3b2287a051d4a8881adb4756316aaa1 (table TestTable [id=26087773aad9435a9eef387b8101db47]) I0507 06:33:17.838495 13867 catalog_manager.cc:1066] Initializing Kudu internal certificate authority... I0507 06:33:17.841399 13867 catalog_manager.cc:1075] Loading token signing keys... I0507 06:33:17.842371 13867 catalog_manager.cc:4303] T 00000000000000000000000000000000 P 0c8516a734564649b36e53a7f5184239: Loaded TSK: 0 I0507 06:33:18.323177 13824 master_service.cc:229] Got heartbeat from unknown tserver (permanent_uuid: "86eeb3337bd640599d56daee2302ae67" instance_seqno: 1557210778720826) as {username='slave'} at 127.0.57.194:41839; Asking this server to re-register. I0507 06:33:18.324057 13573 heartbeater.cc:426] Registering TS with master... I0507 06:33:18.324275 13573 heartbeater.cc:475] Master 127.0.57.254:37965 requested a full tablet report, sending... I0507 06:33:18.325691 13824 ts_manager.cc:159] Registered new tserver with Master: 86eeb3337bd640599d56daee2302ae67 (127.0.57.194:37763) I0507 06:33:18.347951 13824 master_service.cc:229] Got heartbeat from unknown tserver (permanent_uuid: "b8748b85a6fb4418a992adbdfb1a0a62" instance_seqno: 1557210779002793) as {username='slave'} at 127.0.57.195:44869; Asking this server to re-register. I0507 06:33:18.348354 13823 master_service.cc:229] Got heartbeat from unknown tserver (permanent_uuid: "89dd992cec6b407ca4e21db5e6aeef70" instance_seqno: 1557210778399881) as {username='slave'} at 127.0.57.193:56833; Asking this server to re-register. I0507 06:33:18.348486 13695 heartbeater.cc:426] Registering TS with master... I0507 06:33:18.348692 13695 heartbeater.cc:475] Master 127.0.57.254:37965 requested a full tablet report, sending... I0507 06:33:18.348927 13450 heartbeater.cc:426] Registering TS with master... I0507 06:33:18.349087 13450 heartbeater.cc:475] Master 127.0.57.254:37965 requested a full tablet report, sending... I0507 06:33:18.349673 13823 ts_manager.cc:159] Registered new tserver with Master: b8748b85a6fb4418a992adbdfb1a0a62 (127.0.57.195:37113) I0507 06:33:18.350081 13824 ts_manager.cc:159] Registered new tserver with Master: 89dd992cec6b407ca4e21db5e6aeef70 (127.0.57.193:40533) I0507 06:33:18.351583 13824 catalog_manager.cc:3616] Sending ChangeConfig:ADD_PEER:NON_VOTER on tablet d3b2287a051d4a8881adb4756316aaa1 (attempt 1) I0507 06:33:18.358806 13407 consensus_queue.cc:206] T d3b2287a051d4a8881adb4756316aaa1 P 89dd992cec6b407ca4e21db5e6aeef70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 39, Committed index: 39, Last appended: 1.39, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 40 OBSOLETE_local: false peers { permanent_uuid: "b8748b85a6fb4418a992adbdfb1a0a62" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 37113 } } peers { permanent_uuid: "89dd992cec6b407ca4e21db5e6aeef70" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 40533 } } peers { permanent_uuid: "86eeb3337bd640599d56daee2302ae67" member_type: NON_VOTER last_known_addr { host: "127.0.57.194" port: 37763 } attrs { promote: true } } W0507 06:33:18.361418 13341 consensus_peers.cc:457] T d3b2287a051d4a8881adb4756316aaa1 P 89dd992cec6b407ca4e21db5e6aeef70 -> Peer 86eeb3337bd640599d56daee2302ae67 (127.0.57.194:37763): Couldn't send request to peer 86eeb3337bd640599d56daee2302ae67. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry. I0507 06:33:18.361445 13652 raft_consensus.cc:1181] T d3b2287a051d4a8881adb4756316aaa1 P b8748b85a6fb4418a992adbdfb1a0a62 [term 1 FOLLOWER]: Refusing update from remote peer 89dd992cec6b407ca4e21db5e6aeef70: Log matching property violated. Preceding OpId in replica: term: 1 index: 39. Preceding OpId from leader: term: 1 index: 40. (index mismatch) I0507 06:33:18.362009 13783 consensus_queue.cc:984] T d3b2287a051d4a8881adb4756316aaa1 P 89dd992cec6b407ca4e21db5e6aeef70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8748b85a6fb4418a992adbdfb1a0a62" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 37113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 40, Last known committed idx: 39, Time since last communication: 0.000s I0507 06:33:18.364657 13878 raft_consensus.cc:2788] T d3b2287a051d4a8881adb4756316aaa1 P 89dd992cec6b407ca4e21db5e6aeef70 [term 1 LEADER]: Committing config change with OpId 1.40: config changed from index 39 to 40, NON_VOTER 86eeb3337bd640599d56daee2302ae67 (127.0.57.194) added. New config: { opid_index: 40 OBSOLETE_local: false peers { permanent_uuid: "b8748b85a6fb4418a992adbdfb1a0a62" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 37113 } } peers { permanent_uuid: "89dd992cec6b407ca4e21db5e6aeef70" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 40533 } } peers { permanent_uuid: "86eeb3337bd640599d56daee2302ae67" member_type: NON_VOTER last_known_addr { host: "127.0.57.194" port: 37763 } attrs { promote: true } } } I0507 06:33:18.365209 13652 raft_consensus.cc:2788] T d3b2287a051d4a8881adb4756316aaa1 P b8748b85a6fb4418a992adbdfb1a0a62 [term 1 FOLLOWER]: Committing config change with OpId 1.40: config changed from index 39 to 40, NON_VOTER 86eeb3337bd640599d56daee2302ae67 (127.0.57.194) added. New config: { opid_index: 40 OBSOLETE_local: false peers { permanent_uuid: "b8748b85a6fb4418a992adbdfb1a0a62" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 37113 } } peers { permanent_uuid: "89dd992cec6b407ca4e21db5e6aeef70" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 40533 } } peers { permanent_uuid: "86eeb3337bd640599d56daee2302ae67" member_type: NON_VOTER last_known_addr { host: "127.0.57.194" port: 37763 } attrs { promote: true } } } I0507 06:33:18.367609 13806 catalog_manager.cc:3531] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet d3b2287a051d4a8881adb4756316aaa1 with cas_config_opid_index 39: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1) I0507 06:33:18.369241 13824 catalog_manager.cc:3988] T d3b2287a051d4a8881adb4756316aaa1 P b8748b85a6fb4418a992adbdfb1a0a62 reported cstate change: config changed from index 39 to 40, NON_VOTER 86eeb3337bd640599d56daee2302ae67 (127.0.57.194) added. New cstate: current_term: 1 leader_uuid: "89dd992cec6b407ca4e21db5e6aeef70" committed_config { opid_index: 40 OBSOLETE_local: false peers { permanent_uuid: "b8748b85a6fb4418a992adbdfb1a0a62" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 37113 } } peers { permanent_uuid: "89dd992cec6b407ca4e21db5e6aeef70" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 40533 } } peers { permanent_uuid: "86eeb3337bd640599d56daee2302ae67" member_type: NON_VOTER last_known_addr { host: "127.0.57.194" port: 37763 } attrs { promote: true } } } I0507 06:33:18.800616 13885 ts_tablet_manager.cc:676] T d3b2287a051d4a8881adb4756316aaa1 P 86eeb3337bd640599d56daee2302ae67: Initiating tablet copy from peer 89dd992cec6b407ca4e21db5e6aeef70 (127.0.57.193:40533) I0507 06:33:18.801038 13885 tablet_copy_client.cc:240] T d3b2287a051d4a8881adb4756316aaa1 P 86eeb3337bd640599d56daee2302ae67: tablet copy: Beginning tablet copy session from remote peer at address 127.0.57.193:40533 I0507 06:33:18.802085 13427 tablet_copy_service.cc:135] P 89dd992cec6b407ca4e21db5e6aeef70: Received BeginTabletCopySession request for tablet d3b2287a051d4a8881adb4756316aaa1 from peer 86eeb3337bd640599d56daee2302ae67 ({username='slave'} at 127.0.57.194:43523) I0507 06:33:18.802300 13427 tablet_copy_service.cc:156] P 89dd992cec6b407ca4e21db5e6aeef70: Beginning new tablet copy session on tablet d3b2287a051d4a8881adb4756316aaa1 from peer 86eeb3337bd640599d56daee2302ae67 at {username='slave'} at 127.0.57.194:43523: session id = 86eeb3337bd640599d56daee2302ae67-d3b2287a051d4a8881adb4756316aaa1 I0507 06:33:18.805070 13427 tablet_copy_source_session.cc:202] T d3b2287a051d4a8881adb4756316aaa1 P 89dd992cec6b407ca4e21db5e6aeef70: Tablet Copy: opened 5 blocks and 1 log segments I0507 06:33:18.807020 13885 ts_tablet_manager.cc:1377] T d3b2287a051d4a8881adb4756316aaa1 P 86eeb3337bd640599d56daee2302ae67: Deleting tablet data with delete state TABLET_DATA_COPYING I0507 06:33:18.810890 13885 ts_tablet_manager.cc:1390] T d3b2287a051d4a8881adb4756316aaa1 P 86eeb3337bd640599d56daee2302ae67: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.1 I0507 06:33:18.811132 13885 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet d3b2287a051d4a8881adb4756316aaa1. 1 dirs total, 0 dirs full, 0 dirs failed I0507 06:33:18.813283 13885 tablet_copy_client.cc:575] T d3b2287a051d4a8881adb4756316aaa1 P 86eeb3337bd640599d56daee2302ae67: tablet copy: Starting download of 5 data blocks... I0507 06:33:18.823066 13885 tablet_copy_client.cc:538] T d3b2287a051d4a8881adb4756316aaa1 P 86eeb3337bd640599d56daee2302ae67: tablet copy: Starting download of 1 WAL segments... I0507 06:33:18.826596 13885 tablet_copy_client.cc:414] T d3b2287a051d4a8881adb4756316aaa1 P 86eeb3337bd640599d56daee2302ae67: tablet copy: Tablet Copy complete. Replacing tablet superblock. I0507 06:33:18.830245 13885 tablet_bootstrap.cc:439] T d3b2287a051d4a8881adb4756316aaa1 P 86eeb3337bd640599d56daee2302ae67: Bootstrap starting. I0507 06:33:18.845386 13885 tablet_bootstrap.cc:439] T d3b2287a051d4a8881adb4756316aaa1 P 86eeb3337bd640599d56daee2302ae67: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=2. Pending: 0 replicates I0507 06:33:18.845796 13885 tablet_bootstrap.cc:439] T d3b2287a051d4a8881adb4756316aaa1 P 86eeb3337bd640599d56daee2302ae67: Bootstrap complete. I0507 06:33:18.846067 13885 ts_tablet_manager.cc:1053] T d3b2287a051d4a8881adb4756316aaa1 P 86eeb3337bd640599d56daee2302ae67: Time spent bootstrapping tablet: real 0.016s user 0.012s sys 0.004s I0507 06:33:18.846949 13885 raft_consensus.cc:340] T d3b2287a051d4a8881adb4756316aaa1 P 86eeb3337bd640599d56daee2302ae67 [term 1 LEARNER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: 40 OBSOLETE_local: false peers { permanent_uuid: "b8748b85a6fb4418a992adbdfb1a0a62" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 37113 } } peers { permanent_uuid: "89dd992cec6b407ca4e21db5e6aeef70" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 40533 } } peers { permanent_uuid: "86eeb3337bd640599d56daee2302ae67" member_type: NON_VOTER last_known_addr { host: "127.0.57.194" port: 37763 } attrs { promote: true } } I0507 06:33:18.847157 13885 raft_consensus.cc:705] T d3b2287a051d4a8881adb4756316aaa1 P 86eeb3337bd640599d56daee2302ae67 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 86eeb3337bd640599d56daee2302ae67, State: Initialized, Role: LEARNER I0507 06:33:18.847347 13885 consensus_queue.cc:229] T d3b2287a051d4a8881adb4756316aaa1 P 86eeb3337bd640599d56daee2302ae67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 40, Last appended: 1.40, Last appended by leader: 40, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 40 OBSOLETE_local: false peers { permanent_uuid: "b8748b85a6fb4418a992adbdfb1a0a62" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 37113 } } peers { permanent_uuid: "89dd992cec6b407ca4e21db5e6aeef70" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 40533 } } peers { permanent_uuid: "86eeb3337bd640599d56daee2302ae67" member_type: NON_VOTER last_known_addr { host: "127.0.57.194" port: 37763 } attrs { promote: true } } I0507 06:33:18.848104 13885 ts_tablet_manager.cc:1081] T d3b2287a051d4a8881adb4756316aaa1 P 86eeb3337bd640599d56daee2302ae67: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s I0507 06:33:18.849200 13427 tablet_copy_service.cc:325] P 89dd992cec6b407ca4e21db5e6aeef70: Request end of tablet copy session 86eeb3337bd640599d56daee2302ae67-d3b2287a051d4a8881adb4756316aaa1 received from {username='slave'} at 127.0.57.194:43523 I0507 06:33:18.849337 13427 tablet_copy_service.cc:413] P 89dd992cec6b407ca4e21db5e6aeef70: ending tablet copy session 86eeb3337bd640599d56daee2302ae67-d3b2287a051d4a8881adb4756316aaa1 on tablet d3b2287a051d4a8881adb4756316aaa1 with peer 86eeb3337bd640599d56daee2302ae67 I0507 06:33:18.914129 13574 maintenance_manager.cc:306] P 86eeb3337bd640599d56daee2302ae67: Scheduling UndoDeltaBlockGCOp(d3b2287a051d4a8881adb4756316aaa1): 550 bytes on disk I0507 06:33:18.915630 13468 maintenance_manager.cc:501] P 86eeb3337bd640599d56daee2302ae67: UndoDeltaBlockGCOp(d3b2287a051d4a8881adb4756316aaa1) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":91,"lbm_reads_lt_1ms":4} I0507 06:33:19.225903 13530 raft_consensus.cc:1126] T d3b2287a051d4a8881adb4756316aaa1 P 86eeb3337bd640599d56daee2302ae67 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.39->[1.40-1.40] Dedup: 1.40->[] I0507 06:33:19.781296 13881 raft_consensus.cc:974] T d3b2287a051d4a8881adb4756316aaa1 P 89dd992cec6b407ca4e21db5e6aeef70: attempting to promote NON_VOTER 86eeb3337bd640599d56daee2302ae67 to VOTER I0507 06:33:19.781960 13881 consensus_queue.cc:206] T d3b2287a051d4a8881adb4756316aaa1 P 89dd992cec6b407ca4e21db5e6aeef70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 40, Committed index: 40, Last appended: 1.40, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 41 OBSOLETE_local: false peers { permanent_uuid: "b8748b85a6fb4418a992adbdfb1a0a62" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 37113 } } peers { permanent_uuid: "89dd992cec6b407ca4e21db5e6aeef70" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 40533 } } peers { permanent_uuid: "86eeb3337bd640599d56daee2302ae67" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 37763 } attrs { promote: false } } I0507 06:33:19.784001 13530 raft_consensus.cc:1181] T d3b2287a051d4a8881adb4756316aaa1 P 86eeb3337bd640599d56daee2302ae67 [term 1 LEARNER]: Refusing update from remote peer 89dd992cec6b407ca4e21db5e6aeef70: Log matching property violated. Preceding OpId in replica: term: 1 index: 40. Preceding OpId from leader: term: 1 index: 41. (index mismatch) I0507 06:33:19.784621 13652 raft_consensus.cc:1181] T d3b2287a051d4a8881adb4756316aaa1 P b8748b85a6fb4418a992adbdfb1a0a62 [term 1 FOLLOWER]: Refusing update from remote peer 89dd992cec6b407ca4e21db5e6aeef70: Log matching property violated. Preceding OpId in replica: term: 1 index: 40. Preceding OpId from leader: term: 1 index: 41. (index mismatch) I0507 06:33:19.784801 13889 consensus_queue.cc:984] T d3b2287a051d4a8881adb4756316aaa1 P 89dd992cec6b407ca4e21db5e6aeef70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "86eeb3337bd640599d56daee2302ae67" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 37763 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 41, Last known committed idx: 40, Time since last communication: 0.000s I0507 06:33:19.785223 13891 consensus_queue.cc:984] T d3b2287a051d4a8881adb4756316aaa1 P 89dd992cec6b407ca4e21db5e6aeef70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8748b85a6fb4418a992adbdfb1a0a62" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 37113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 41, Last known committed idx: 40, Time since last communication: 0.000s I0507 06:33:19.789968 13889 raft_consensus.cc:2788] T d3b2287a051d4a8881adb4756316aaa1 P 89dd992cec6b407ca4e21db5e6aeef70 [term 1 LEADER]: Committing config change with OpId 1.41: config changed from index 40 to 41, 86eeb3337bd640599d56daee2302ae67 (127.0.57.194) changed from NON_VOTER to VOTER. New config: { opid_index: 41 OBSOLETE_local: false peers { permanent_uuid: "b8748b85a6fb4418a992adbdfb1a0a62" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 37113 } } peers { permanent_uuid: "89dd992cec6b407ca4e21db5e6aeef70" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 40533 } } peers { permanent_uuid: "86eeb3337bd640599d56daee2302ae67" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 37763 } attrs { promote: false } } } I0507 06:33:19.790655 13652 raft_consensus.cc:2788] T d3b2287a051d4a8881adb4756316aaa1 P b8748b85a6fb4418a992adbdfb1a0a62 [term 1 FOLLOWER]: Committing config change with OpId 1.41: config changed from index 40 to 41, 86eeb3337bd640599d56daee2302ae67 (127.0.57.194) changed from NON_VOTER to VOTER. New config: { opid_index: 41 OBSOLETE_local: false peers { permanent_uuid: "b8748b85a6fb4418a992adbdfb1a0a62" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 37113 } } peers { permanent_uuid: "89dd992cec6b407ca4e21db5e6aeef70" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 40533 } } peers { permanent_uuid: "86eeb3337bd640599d56daee2302ae67" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 37763 } attrs { promote: false } } } I0507 06:33:19.791075 13530 raft_consensus.cc:2788] T d3b2287a051d4a8881adb4756316aaa1 P 86eeb3337bd640599d56daee2302ae67 [term 1 FOLLOWER]: Committing config change with OpId 1.41: config changed from index 40 to 41, 86eeb3337bd640599d56daee2302ae67 (127.0.57.194) changed from NON_VOTER to VOTER. New config: { opid_index: 41 OBSOLETE_local: false peers { permanent_uuid: "b8748b85a6fb4418a992adbdfb1a0a62" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 37113 } } peers { permanent_uuid: "89dd992cec6b407ca4e21db5e6aeef70" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 40533 } } peers { permanent_uuid: "86eeb3337bd640599d56daee2302ae67" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 37763 } attrs { promote: false } } } I0507 06:33:19.794106 13823 catalog_manager.cc:3988] T d3b2287a051d4a8881adb4756316aaa1 P 89dd992cec6b407ca4e21db5e6aeef70 reported cstate change: config changed from index 40 to 41, 86eeb3337bd640599d56daee2302ae67 (127.0.57.194) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "89dd992cec6b407ca4e21db5e6aeef70" committed_config { opid_index: 41 OBSOLETE_local: false peers { permanent_uuid: "b8748b85a6fb4418a992adbdfb1a0a62" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 37113 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "89dd992cec6b407ca4e21db5e6aeef70" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 40533 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "86eeb3337bd640599d56daee2302ae67" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 37763 } attrs { promote: false } health_report { overall_health: HEALTHY } } } Master Summary UUID | Address | Status ----------------------------------+--------------------+--------- 0c8516a734564649b36e53a7f5184239 | 127.0.57.254:37965 | HEALTHY Flag | Value | Tags | Master ------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+------------------------- ipki_ca_key_size | 1024 | experimental | all 1 server(s) checked ipki_server_key_size | 1024 | experimental | all 1 server(s) checked master_add_server_when_underreplicated | true | hidden | all 1 server(s) checked never_fsync | true | unsafe,advanced | all 1 server(s) checked raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked rpc_reuseport | true | experimental | all 1 server(s) checked rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked server_dump_info_format | pb | hidden | all 1 server(s) checked server_dump_info_path | /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_1.1557210500371863-231/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked Tablet Server Summary UUID | Address | Status | Location ----------------------------------+--------------------+---------+---------- 86eeb3337bd640599d56daee2302ae67 | 127.0.57.194:37763 | HEALTHY | 89dd992cec6b407ca4e21db5e6aeef70 | 127.0.57.193:40533 | HEALTHY | b8748b85a6fb4418a992adbdfb1a0a62 | 127.0.57.195:37113 | HEALTHY | Tablet Server Location Summary Location | Count ----------+--------- | 3 Flag | Value | Tags | Tablet Server ------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------------------------------+------------------------- ipki_server_key_size | 1024 | experimental | all 3 server(s) checked local_ip_for_outbound_sockets | 127.0.57.193 | experimental | 127.0.57.193:40533 local_ip_for_outbound_sockets | 127.0.57.194 | experimental | 127.0.57.194:37763 local_ip_for_outbound_sockets | 127.0.57.195 | experimental | 127.0.57.195:37113 log_compression_codec | none | experimental | all 3 server(s) checked log_max_segments_to_retain | 3 | experimental,advanced,runtime | all 3 server(s) checked log_target_replay_size_mb | 1 | experimental | all 3 server(s) checked maintenance_manager_polling_interval_ms | 100 | hidden | all 3 server(s) checked max_cell_size_bytes | 1000000 | unsafe | all 3 server(s) checked never_fsync | true | unsafe,advanced | all 3 server(s) checked raft_prepare_replacement_before_eviction | true | experimental,advanced | all 3 server(s) checked rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked server_dump_info_format | pb | hidden | all 3 server(s) checked server_dump_info_path | /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_1.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.0.57.193:40533 server_dump_info_path | /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_1.1557210500371863-231/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.0.57.194:37763 server_dump_info_path | /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_1.1557210500371863-231/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.0.57.195:37113 Version Summary Version | Servers ------------------------+------------------------- 1.10.0-cdh6.x-SNAPSHOT | all 4 server(s) checked Tablet Summary Summary by table Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable -----------+----+---------+---------------+---------+------------+------------------+------------- TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0 Tablet Replica Count Summary Statistic | Replica Count ----------------+--------------- Minimum | 1 First Quartile | 1 Median | 1 Third Quartile | 1 Maximum | 1 Total Count Summary | Total Count ----------------+------------- Masters | 1 Tablet Servers | 3 Tables | 1 Tablets | 1 Replicas | 3 ================== Warnings: ================== Some masters have unsafe, experimental, or hidden flags set Some tablet servers have unsafe, experimental, or hidden flags set OK I0507 06:33:19.889016 231 log_verifier.cc:121] Checking tablet d3b2287a051d4a8881adb4756316aaa1 I0507 06:33:19.901211 231 log_verifier.cc:172] Verified matching terms for 9 ops in tablet d3b2287a051d4a8881adb4756316aaa1 I0507 06:33:19.902281 231 external_mini_cluster.cc:1172] Attempting to check leaks for /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver pid 13320 I0507 06:33:19.903466 13447 generic_service.cc:171] Checking for leaks (request via RPC) I0507 06:33:20.002512 231 external_mini_cluster.cc:1131] Killing /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver with pid 13320 I0507 06:33:20.019639 231 external_mini_cluster.cc:1172] Attempting to check leaks for /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver pid 13454 I0507 06:33:20.020769 13570 generic_service.cc:171] Checking for leaks (request via RPC) I0507 06:33:20.084098 231 external_mini_cluster.cc:1131] Killing /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver with pid 13454 I0507 06:33:20.096556 231 external_mini_cluster.cc:1172] Attempting to check leaks for /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver pid 13576 I0507 06:33:20.097764 13692 generic_service.cc:171] Checking for leaks (request via RPC) I0507 06:33:20.184811 231 external_mini_cluster.cc:1131] Killing /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver with pid 13576 I0507 06:33:20.201099 231 external_mini_cluster.cc:1172] Attempting to check leaks for /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-master pid 13798 I0507 06:33:20.202229 13854 generic_service.cc:171] Checking for leaks (request via RPC) I0507 06:33:20.295655 231 external_mini_cluster.cc:1131] Killing /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-master with pid 13798 [ OK ] ReplicaBehindWalGcThresholdITest.ReplicaReplacement/1 (22640 ms) [ RUN ] ReplicaBehindWalGcThresholdITest.ReplicaReplacement/2 I0507 06:33:20.324968 231 test_util.cc:212] Using random seed: 1687683411 I0507 06:33:20.326692 231 ts_itest-base.cc:112] Starting cluster with: I0507 06:33:20.326733 231 ts_itest-base.cc:113] -------------- I0507 06:33:20.326762 231 ts_itest-base.cc:114] 3 tablet servers I0507 06:33:20.326800 231 ts_itest-base.cc:115] 3 replicas per TS I0507 06:33:20.326828 231 ts_itest-base.cc:116] -------------- I0507 06:33:20.327996 231 external_mini_cluster.cc:933] Running /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-master /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-master --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_2.1557210500371863-231/raft_consensus-itest-cluster/master-0/wal --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_2.1557210500371863-231/raft_consensus-itest-cluster/master-0/data --block_manager=log --webserver_interface=localhost --ipki_ca_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.0.57.254:44949 --webserver_interface=127.0.57.254 --webserver_port=0 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_2.1557210500371863-231/raft_consensus-itest-cluster/master-0/logs --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_2.1557210500371863-231/raft_consensus-itest-cluster/master-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=true --master_add_server_when_underreplicated=false --rpc_reuseport=true with env {} WARNING: Logging before InitGoogleLogging() is written to STDERR W0507 06:33:20.550662 13935 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0507 06:33:20.550921 13935 flags.cc:404] Enabled unsafe flag: --never_fsync=true W0507 06:33:20.553645 13935 flags.cc:404] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true W0507 06:33:20.553716 13935 flags.cc:404] Enabled experimental flag: --ipki_ca_key_size=1024 W0507 06:33:20.553791 13935 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W0507 06:33:20.553818 13935 flags.cc:404] Enabled experimental flag: --tsk_num_rsa_bits=512 W0507 06:33:20.553881 13935 flags.cc:404] Enabled experimental flag: --rpc_reuseport=true I0507 06:33:20.559095 13935 master_main.cc:73] Master server non-default flags: --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_2.1557210500371863-231/raft_consensus-itest-cluster/master-0/data --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_2.1557210500371863-231/raft_consensus-itest-cluster/master-0/wal --master_add_server_when_underreplicated=false --ipki_ca_key_size=1024 --ipki_server_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.0.57.254:44949 --rpc_reuseport=true --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_2.1557210500371863-231/raft_consensus-itest-cluster/master-0/data/info.pb --webserver_interface=127.0.57.254 --webserver_port=0 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_2.1557210500371863-231/raft_consensus-itest-cluster/master-0/logs --logbuflevel=-1 --logtostderr=true Master server version: kudu 1.10.0-cdh6.x-SNAPSHOT revision 7d0ee26697926e87af9ef395008799577f6ab43f build type FASTDEBUG built by jenkins at 06 May 2019 22:44:21 PST on quasar-knmpfx-1.vpc.cloudera.com build id 322017 ASAN enabled I0507 06:33:20.566416 13935 system_ntp.cc:202] NTP initialized. Skew: 500ppm Current error: 174516us I0507 06:33:20.567878 13935 fs_manager.cc:263] Metadata directory not provided I0507 06:33:20.567953 13935 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0507 06:33:20.568325 13935 server_base.cc:433] Could not load existing FS layout: Not found: could not find a healthy instance file I0507 06:33:20.568399 13935 server_base.cc:434] Attempting to create new FS layout instead I0507 06:33:20.571259 13935 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_2.1557210500371863-231/raft_consensus-itest-cluster/master-0/data/instance: uuid: "16571e2dbc9d46159c1792ce321cd0d7" format_stamp: "Formatted at 2019-05-07 06:33:20 on dist-test-slave-dist-test-slave-rqp6" I0507 06:33:20.571863 13935 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_2.1557210500371863-231/raft_consensus-itest-cluster/master-0/wal/instance: uuid: "16571e2dbc9d46159c1792ce321cd0d7" format_stamp: "Formatted at 2019-05-07 06:33:20 on dist-test-slave-dist-test-slave-rqp6" I0507 06:33:20.579934 13935 fs_manager.cc:503] Time spent creating directory manager: real 0.008s user 0.008s sys 0.000s I0507 06:33:20.581241 13935 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0507 06:33:20.581902 13935 file_cache.cc:466] Constructed file cache lbm with capacity 419430 I0507 06:33:20.585824 13935 fs_manager.cc:419] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s I0507 06:33:20.585922 13935 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_2.1557210500371863-231/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_2.1557210500371863-231/raft_consensus-itest-cluster/master-0/wal uuid: "16571e2dbc9d46159c1792ce321cd0d7" format_stamp: "Formatted at 2019-05-07 06:33:20 on dist-test-slave-dist-test-slave-rqp6" I0507 06:33:20.586102 13935 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_2.1557210500371863-231/raft_consensus-itest-cluster/master-0/wal metadata directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_2.1557210500371863-231/raft_consensus-itest-cluster/master-0/wal 1 data directories: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_2.1557210500371863-231/raft_consensus-itest-cluster/master-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0507 06:33:20.630894 13935 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0507 06:33:20.645684 13935 rpc_server.cc:205] RPC server started. Bound to: 127.0.57.254:44949 I0507 06:33:20.646965 13935 webserver.cc:308] Webserver started at http://127.0.57.254:33821/ using document root and password file I0507 06:33:20.647794 13935 server_base.cc:608] Dumped server information to /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_2.1557210500371863-231/raft_consensus-itest-cluster/master-0/data/info.pb I0507 06:33:20.651726 13994 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0507 06:33:20.651736 231 external_mini_cluster.cc:995] Started /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-master as pid 13935 I0507 06:33:20.652412 231 external_mini_cluster.cc:933] Running /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_2.1557210500371863-231/raft_consensus-itest-cluster/ts-0/wal --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_2.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data --block_manager=log --rpc_bind_addresses=127.0.57.193:0 --local_ip_for_outbound_sockets=127.0.57.193 --webserver_interface=127.0.57.193 --webserver_port=0 --tserver_master_addrs=127.0.57.254:44949 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_2.1557210500371863-231/raft_consensus-itest-cluster/ts-0/logs --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_2.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=true --follower_unavailable_considered_failed_sec=5 --log_compression_codec=none --log_cache_size_limit_mb=1 --log_segment_size_mb=1 --log_async_preallocate_segments=false --log_min_segments_to_retain=1 --log_max_segments_to_retain=3 --maintenance_manager_polling_interval_ms=100 --log_target_replay_size_mb=1 --max_cell_size_bytes=1000000 with env {} I0507 06:33:20.664248 13994 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 16571e2dbc9d46159c1792ce321cd0d7: Bootstrap starting. I0507 06:33:20.667527 13994 tablet_bootstrap.cc:590] T 00000000000000000000000000000000 P 16571e2dbc9d46159c1792ce321cd0d7: No blocks or log segments found. Creating new log. I0507 06:33:20.668623 13994 log.cc:525] T 00000000000000000000000000000000 P 16571e2dbc9d46159c1792ce321cd0d7: Log is configured to *not* fsync() on all Append() calls I0507 06:33:20.674268 13994 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 16571e2dbc9d46159c1792ce321cd0d7: No bootstrap required, opened a new log I0507 06:33:20.681643 13994 raft_consensus.cc:340] T 00000000000000000000000000000000 P 16571e2dbc9d46159c1792ce321cd0d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16571e2dbc9d46159c1792ce321cd0d7" member_type: VOTER } I0507 06:33:20.681913 13994 raft_consensus.cc:366] T 00000000000000000000000000000000 P 16571e2dbc9d46159c1792ce321cd0d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0507 06:33:20.681978 13994 raft_consensus.cc:705] T 00000000000000000000000000000000 P 16571e2dbc9d46159c1792ce321cd0d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16571e2dbc9d46159c1792ce321cd0d7, State: Initialized, Role: FOLLOWER I0507 06:33:20.682565 13994 consensus_queue.cc:229] T 00000000000000000000000000000000 P 16571e2dbc9d46159c1792ce321cd0d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16571e2dbc9d46159c1792ce321cd0d7" member_type: VOTER } I0507 06:33:20.682745 13994 raft_consensus.cc:380] T 00000000000000000000000000000000 P 16571e2dbc9d46159c1792ce321cd0d7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0507 06:33:20.682823 13994 raft_consensus.cc:472] T 00000000000000000000000000000000 P 16571e2dbc9d46159c1792ce321cd0d7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0507 06:33:20.682932 13994 raft_consensus.cc:2886] T 00000000000000000000000000000000 P 16571e2dbc9d46159c1792ce321cd0d7 [term 0 FOLLOWER]: Advancing to term 1 I0507 06:33:20.684792 13994 raft_consensus.cc:494] T 00000000000000000000000000000000 P 16571e2dbc9d46159c1792ce321cd0d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16571e2dbc9d46159c1792ce321cd0d7" member_type: VOTER } I0507 06:33:20.685281 13994 leader_election.cc:310] T 00000000000000000000000000000000 P 16571e2dbc9d46159c1792ce321cd0d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 16571e2dbc9d46159c1792ce321cd0d7; no voters: I0507 06:33:20.689100 13994 leader_election.cc:296] T 00000000000000000000000000000000 P 16571e2dbc9d46159c1792ce321cd0d7 [CANDIDATE]: Term 1 election: Requested vote from peers I0507 06:33:20.689529 13997 raft_consensus.cc:2640] T 00000000000000000000000000000000 P 16571e2dbc9d46159c1792ce321cd0d7 [term 1 FOLLOWER]: Leader election won for term 1 I0507 06:33:20.690376 13997 raft_consensus.cc:667] T 00000000000000000000000000000000 P 16571e2dbc9d46159c1792ce321cd0d7 [term 1 LEADER]: Becoming Leader. State: Replica: 16571e2dbc9d46159c1792ce321cd0d7, State: Running, Role: LEADER I0507 06:33:20.690918 13997 consensus_queue.cc:206] T 00000000000000000000000000000000 P 16571e2dbc9d46159c1792ce321cd0d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16571e2dbc9d46159c1792ce321cd0d7" member_type: VOTER } I0507 06:33:20.697903 13994 sys_catalog.cc:337] T 00000000000000000000000000000000 P 16571e2dbc9d46159c1792ce321cd0d7 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 1 leader_uuid: "16571e2dbc9d46159c1792ce321cd0d7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16571e2dbc9d46159c1792ce321cd0d7" member_type: VOTER } } I0507 06:33:20.698025 13998 sys_catalog.cc:337] T 00000000000000000000000000000000 P 16571e2dbc9d46159c1792ce321cd0d7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 16571e2dbc9d46159c1792ce321cd0d7. Latest consensus state: current_term: 1 leader_uuid: "16571e2dbc9d46159c1792ce321cd0d7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16571e2dbc9d46159c1792ce321cd0d7" member_type: VOTER } } I0507 06:33:20.698276 13998 sys_catalog.cc:340] T 00000000000000000000000000000000 P 16571e2dbc9d46159c1792ce321cd0d7 [sys.catalog]: This master's current role is: LEADER I0507 06:33:20.698298 13994 sys_catalog.cc:340] T 00000000000000000000000000000000 P 16571e2dbc9d46159c1792ce321cd0d7 [sys.catalog]: This master's current role is: LEADER I0507 06:33:20.698861 13999 sys_catalog.cc:337] T 00000000000000000000000000000000 P 16571e2dbc9d46159c1792ce321cd0d7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "16571e2dbc9d46159c1792ce321cd0d7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16571e2dbc9d46159c1792ce321cd0d7" member_type: VOTER } } I0507 06:33:20.699098 13999 sys_catalog.cc:340] T 00000000000000000000000000000000 P 16571e2dbc9d46159c1792ce321cd0d7 [sys.catalog]: This master's current role is: LEADER I0507 06:33:20.700268 13994 sys_catalog.cc:424] T 00000000000000000000000000000000 P 16571e2dbc9d46159c1792ce321cd0d7 [sys.catalog]: configured and running, proceeding with master startup. W0507 06:33:20.705106 14003 catalog_manager.cc:1113] T 00000000000000000000000000000000 P 16571e2dbc9d46159c1792ce321cd0d7: acquiring CA information for follower catalog manager: Not found: root CA entry not found W0507 06:33:20.705204 14003 catalog_manager.cc:596] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry I0507 06:33:20.705277 14001 catalog_manager.cc:1055] Loading table and tablet metadata into memory... I0507 06:33:20.707424 14001 catalog_manager.cc:1066] Initializing Kudu internal certificate authority... I0507 06:33:20.731070 14001 catalog_manager.cc:950] Generated new certificate authority record I0507 06:33:20.732434 14001 catalog_manager.cc:1075] Loading token signing keys... I0507 06:33:20.743501 14001 catalog_manager.cc:4292] T 00000000000000000000000000000000 P 16571e2dbc9d46159c1792ce321cd0d7: Generated new TSK 0 WARNING: Logging before InitGoogleLogging() is written to STDERR W0507 06:33:20.880239 13995 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0507 06:33:20.880614 13995 flags.cc:404] Enabled unsafe flag: --max_cell_size_bytes=1000000 W0507 06:33:20.880735 13995 flags.cc:404] Enabled unsafe flag: --never_fsync=true W0507 06:33:20.884094 13995 flags.cc:404] Enabled experimental flag: --log_compression_codec=none W0507 06:33:20.884186 13995 flags.cc:404] Enabled experimental flag: --log_max_segments_to_retain=3 W0507 06:33:20.884258 13995 flags.cc:404] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true W0507 06:33:20.884335 13995 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W0507 06:33:20.884490 13995 flags.cc:404] Enabled experimental flag: --log_target_replay_size_mb=1 W0507 06:33:20.884604 13995 flags.cc:404] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.57.193 I0507 06:33:20.892547 13995 tablet_server_main.cc:78] Tablet server non-default flags: --follower_unavailable_considered_failed_sec=5 --log_compression_codec=none --log_max_segments_to_retain=3 --log_cache_size_limit_mb=1 --log_async_preallocate_segments=false --log_segment_size_mb=1 --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_2.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_2.1557210500371863-231/raft_consensus-itest-cluster/ts-0/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.0.57.193:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_2.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data/info.pb --webserver_interface=127.0.57.193 --webserver_port=0 --max_cell_size_bytes=1000000 --tserver_master_addrs=127.0.57.254:44949 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --log_target_replay_size_mb=1 --maintenance_manager_polling_interval_ms=100 --enable_minidumps=false --local_ip_for_outbound_sockets=127.0.57.193 --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_2.1557210500371863-231/raft_consensus-itest-cluster/ts-0/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.10.0-cdh6.x-SNAPSHOT revision 7d0ee26697926e87af9ef395008799577f6ab43f build type FASTDEBUG built by jenkins at 06 May 2019 22:44:21 PST on quasar-knmpfx-1.vpc.cloudera.com build id 322017 ASAN enabled I0507 06:33:20.903463 13995 system_ntp.cc:202] NTP initialized. Skew: 500ppm Current error: 174516us I0507 06:33:20.904404 13995 fs_manager.cc:263] Metadata directory not provided I0507 06:33:20.904489 13995 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0507 06:33:20.904913 13995 server_base.cc:433] Could not load existing FS layout: Not found: could not find a healthy instance file I0507 06:33:20.905006 13995 server_base.cc:434] Attempting to create new FS layout instead I0507 06:33:20.908324 13995 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_2.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data/instance: uuid: "784329b2a1db4d4cb2e93c98aea55809" format_stamp: "Formatted at 2019-05-07 06:33:20 on dist-test-slave-dist-test-slave-rqp6" I0507 06:33:20.909145 13995 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_2.1557210500371863-231/raft_consensus-itest-cluster/ts-0/wal/instance: uuid: "784329b2a1db4d4cb2e93c98aea55809" format_stamp: "Formatted at 2019-05-07 06:33:20 on dist-test-slave-dist-test-slave-rqp6" I0507 06:33:20.919319 13995 fs_manager.cc:503] Time spent creating directory manager: real 0.010s user 0.005s sys 0.004s I0507 06:33:20.920848 13995 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0507 06:33:20.921720 13995 file_cache.cc:466] Constructed file cache lbm with capacity 419430 I0507 06:33:20.926113 13995 fs_manager.cc:419] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s I0507 06:33:20.926250 13995 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_2.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_2.1557210500371863-231/raft_consensus-itest-cluster/ts-0/wal uuid: "784329b2a1db4d4cb2e93c98aea55809" format_stamp: "Formatted at 2019-05-07 06:33:20 on dist-test-slave-dist-test-slave-rqp6" I0507 06:33:20.926461 13995 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_2.1557210500371863-231/raft_consensus-itest-cluster/ts-0/wal metadata directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_2.1557210500371863-231/raft_consensus-itest-cluster/ts-0/wal 1 data directories: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_2.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0507 06:33:21.009883 13995 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0507 06:33:21.012403 13995 ts_tablet_manager.cc:356] Loaded tablet metadata (0 total tablets, 0 live tablets) I0507 06:33:21.012504 13995 ts_tablet_manager.cc:376] Registered 0 tablets I0507 06:33:21.119963 13995 rpc_server.cc:205] RPC server started. Bound to: 127.0.57.193:46571 I0507 06:33:21.121309 13995 webserver.cc:308] Webserver started at http://127.0.57.193:35267/ using document root and password file I0507 06:33:21.122054 13995 server_base.cc:608] Dumped server information to /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_2.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data/info.pb I0507 06:33:21.129700 231 external_mini_cluster.cc:995] Started /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver as pid 13995 I0507 06:33:21.130689 231 external_mini_cluster.cc:933] Running /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_2.1557210500371863-231/raft_consensus-itest-cluster/ts-1/wal --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_2.1557210500371863-231/raft_consensus-itest-cluster/ts-1/data --block_manager=log --rpc_bind_addresses=127.0.57.194:0 --local_ip_for_outbound_sockets=127.0.57.194 --webserver_interface=127.0.57.194 --webserver_port=0 --tserver_master_addrs=127.0.57.254:44949 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_2.1557210500371863-231/raft_consensus-itest-cluster/ts-1/logs --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_2.1557210500371863-231/raft_consensus-itest-cluster/ts-1/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=true --follower_unavailable_considered_failed_sec=5 --log_compression_codec=none --log_cache_size_limit_mb=1 --log_segment_size_mb=1 --log_async_preallocate_segments=false --log_min_segments_to_retain=1 --log_max_segments_to_retain=3 --maintenance_manager_polling_interval_ms=100 --log_target_replay_size_mb=1 --max_cell_size_bytes=1000000 with env {} I0507 06:33:21.140763 14124 heartbeater.cc:345] Connected to a master server at 127.0.57.254:44949 I0507 06:33:21.141021 14124 heartbeater.cc:426] Registering TS with master... I0507 06:33:21.141733 14124 heartbeater.cc:475] Master 127.0.57.254:44949 requested a full tablet report, sending... I0507 06:33:21.143749 13961 ts_manager.cc:159] Registered new tserver with Master: 784329b2a1db4d4cb2e93c98aea55809 (127.0.57.193:46571) I0507 06:33:21.145426 13961 master_service.cc:273] Signed X509 certificate for tserver {username='slave'} at 127.0.57.193:54299 WARNING: Logging before InitGoogleLogging() is written to STDERR W0507 06:33:21.298559 14128 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0507 06:33:21.298786 14128 flags.cc:404] Enabled unsafe flag: --max_cell_size_bytes=1000000 W0507 06:33:21.298837 14128 flags.cc:404] Enabled unsafe flag: --never_fsync=true W0507 06:33:21.300884 14128 flags.cc:404] Enabled experimental flag: --log_compression_codec=none W0507 06:33:21.300949 14128 flags.cc:404] Enabled experimental flag: --log_max_segments_to_retain=3 W0507 06:33:21.300997 14128 flags.cc:404] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true W0507 06:33:21.301046 14128 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W0507 06:33:21.301127 14128 flags.cc:404] Enabled experimental flag: --log_target_replay_size_mb=1 W0507 06:33:21.301178 14128 flags.cc:404] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.57.194 I0507 06:33:21.305128 14128 tablet_server_main.cc:78] Tablet server non-default flags: --follower_unavailable_considered_failed_sec=5 --log_compression_codec=none --log_max_segments_to_retain=3 --log_cache_size_limit_mb=1 --log_async_preallocate_segments=false --log_segment_size_mb=1 --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_2.1557210500371863-231/raft_consensus-itest-cluster/ts-1/data --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_2.1557210500371863-231/raft_consensus-itest-cluster/ts-1/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.0.57.194:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_2.1557210500371863-231/raft_consensus-itest-cluster/ts-1/data/info.pb --webserver_interface=127.0.57.194 --webserver_port=0 --max_cell_size_bytes=1000000 --tserver_master_addrs=127.0.57.254:44949 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --log_target_replay_size_mb=1 --maintenance_manager_polling_interval_ms=100 --enable_minidumps=false --local_ip_for_outbound_sockets=127.0.57.194 --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_2.1557210500371863-231/raft_consensus-itest-cluster/ts-1/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.10.0-cdh6.x-SNAPSHOT revision 7d0ee26697926e87af9ef395008799577f6ab43f build type FASTDEBUG built by jenkins at 06 May 2019 22:44:21 PST on quasar-knmpfx-1.vpc.cloudera.com build id 322017 ASAN enabled I0507 06:33:21.313063 14128 system_ntp.cc:202] NTP initialized. Skew: 500ppm Current error: 175016us I0507 06:33:21.313661 14128 fs_manager.cc:263] Metadata directory not provided I0507 06:33:21.313725 14128 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0507 06:33:21.313999 14128 server_base.cc:433] Could not load existing FS layout: Not found: could not find a healthy instance file I0507 06:33:21.314061 14128 server_base.cc:434] Attempting to create new FS layout instead I0507 06:33:21.316228 14128 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_2.1557210500371863-231/raft_consensus-itest-cluster/ts-1/data/instance: uuid: "539eecf299464551a0697eeab25b54c2" format_stamp: "Formatted at 2019-05-07 06:33:21 on dist-test-slave-dist-test-slave-rqp6" I0507 06:33:21.316764 14128 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_2.1557210500371863-231/raft_consensus-itest-cluster/ts-1/wal/instance: uuid: "539eecf299464551a0697eeab25b54c2" format_stamp: "Formatted at 2019-05-07 06:33:21 on dist-test-slave-dist-test-slave-rqp6" I0507 06:33:21.323427 14128 fs_manager.cc:503] Time spent creating directory manager: real 0.006s user 0.004s sys 0.001s I0507 06:33:21.324848 14128 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0507 06:33:21.325531 14128 file_cache.cc:466] Constructed file cache lbm with capacity 419430 I0507 06:33:21.328791 14128 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s I0507 06:33:21.328899 14128 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_2.1557210500371863-231/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_2.1557210500371863-231/raft_consensus-itest-cluster/ts-1/wal uuid: "539eecf299464551a0697eeab25b54c2" format_stamp: "Formatted at 2019-05-07 06:33:21 on dist-test-slave-dist-test-slave-rqp6" I0507 06:33:21.329111 14128 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_2.1557210500371863-231/raft_consensus-itest-cluster/ts-1/wal metadata directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_2.1557210500371863-231/raft_consensus-itest-cluster/ts-1/wal 1 data directories: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_2.1557210500371863-231/raft_consensus-itest-cluster/ts-1/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0507 06:33:21.376334 14128 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0507 06:33:21.378235 14128 ts_tablet_manager.cc:356] Loaded tablet metadata (0 total tablets, 0 live tablets) I0507 06:33:21.378326 14128 ts_tablet_manager.cc:376] Registered 0 tablets I0507 06:33:21.407521 14128 rpc_server.cc:205] RPC server started. Bound to: 127.0.57.194:45599 I0507 06:33:21.408728 14128 webserver.cc:308] Webserver started at http://127.0.57.194:35981/ using document root and password file I0507 06:33:21.409412 14128 server_base.cc:608] Dumped server information to /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_2.1557210500371863-231/raft_consensus-itest-cluster/ts-1/data/info.pb I0507 06:33:21.418406 231 external_mini_cluster.cc:995] Started /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver as pid 14128 I0507 06:33:21.419076 231 external_mini_cluster.cc:933] Running /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_2.1557210500371863-231/raft_consensus-itest-cluster/ts-2/wal --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_2.1557210500371863-231/raft_consensus-itest-cluster/ts-2/data --block_manager=log --rpc_bind_addresses=127.0.57.195:0 --local_ip_for_outbound_sockets=127.0.57.195 --webserver_interface=127.0.57.195 --webserver_port=0 --tserver_master_addrs=127.0.57.254:44949 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_2.1557210500371863-231/raft_consensus-itest-cluster/ts-2/logs --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_2.1557210500371863-231/raft_consensus-itest-cluster/ts-2/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=true --follower_unavailable_considered_failed_sec=5 --log_compression_codec=none --log_cache_size_limit_mb=1 --log_segment_size_mb=1 --log_async_preallocate_segments=false --log_min_segments_to_retain=1 --log_max_segments_to_retain=3 --maintenance_manager_polling_interval_ms=100 --log_target_replay_size_mb=1 --max_cell_size_bytes=1000000 with env {} I0507 06:33:21.423210 14247 heartbeater.cc:345] Connected to a master server at 127.0.57.254:44949 I0507 06:33:21.423427 14247 heartbeater.cc:426] Registering TS with master... I0507 06:33:21.424005 14247 heartbeater.cc:475] Master 127.0.57.254:44949 requested a full tablet report, sending... I0507 06:33:21.425384 13961 ts_manager.cc:159] Registered new tserver with Master: 539eecf299464551a0697eeab25b54c2 (127.0.57.194:45599) I0507 06:33:21.426362 13961 master_service.cc:273] Signed X509 certificate for tserver {username='slave'} at 127.0.57.194:56607 WARNING: Logging before InitGoogleLogging() is written to STDERR W0507 06:33:21.596246 14250 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0507 06:33:21.596493 14250 flags.cc:404] Enabled unsafe flag: --max_cell_size_bytes=1000000 W0507 06:33:21.596590 14250 flags.cc:404] Enabled unsafe flag: --never_fsync=true W0507 06:33:21.598767 14250 flags.cc:404] Enabled experimental flag: --log_compression_codec=none W0507 06:33:21.598831 14250 flags.cc:404] Enabled experimental flag: --log_max_segments_to_retain=3 W0507 06:33:21.598886 14250 flags.cc:404] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true W0507 06:33:21.598958 14250 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W0507 06:33:21.599063 14250 flags.cc:404] Enabled experimental flag: --log_target_replay_size_mb=1 W0507 06:33:21.599130 14250 flags.cc:404] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.57.195 I0507 06:33:21.603323 14250 tablet_server_main.cc:78] Tablet server non-default flags: --follower_unavailable_considered_failed_sec=5 --log_compression_codec=none --log_max_segments_to_retain=3 --log_cache_size_limit_mb=1 --log_async_preallocate_segments=false --log_segment_size_mb=1 --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_2.1557210500371863-231/raft_consensus-itest-cluster/ts-2/data --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_2.1557210500371863-231/raft_consensus-itest-cluster/ts-2/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.0.57.195:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_2.1557210500371863-231/raft_consensus-itest-cluster/ts-2/data/info.pb --webserver_interface=127.0.57.195 --webserver_port=0 --max_cell_size_bytes=1000000 --tserver_master_addrs=127.0.57.254:44949 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --log_target_replay_size_mb=1 --maintenance_manager_polling_interval_ms=100 --enable_minidumps=false --local_ip_for_outbound_sockets=127.0.57.195 --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_2.1557210500371863-231/raft_consensus-itest-cluster/ts-2/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.10.0-cdh6.x-SNAPSHOT revision 7d0ee26697926e87af9ef395008799577f6ab43f build type FASTDEBUG built by jenkins at 06 May 2019 22:44:21 PST on quasar-knmpfx-1.vpc.cloudera.com build id 322017 ASAN enabled I0507 06:33:21.611517 14250 system_ntp.cc:202] NTP initialized. Skew: 500ppm Current error: 175016us I0507 06:33:21.612264 14250 fs_manager.cc:263] Metadata directory not provided I0507 06:33:21.612330 14250 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0507 06:33:21.612735 14250 server_base.cc:433] Could not load existing FS layout: Not found: could not find a healthy instance file I0507 06:33:21.612804 14250 server_base.cc:434] Attempting to create new FS layout instead I0507 06:33:21.615370 14250 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_2.1557210500371863-231/raft_consensus-itest-cluster/ts-2/data/instance: uuid: "e1881b44d0ad42d5b3c8cbe5e38aed5c" format_stamp: "Formatted at 2019-05-07 06:33:21 on dist-test-slave-dist-test-slave-rqp6" I0507 06:33:21.615967 14250 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_2.1557210500371863-231/raft_consensus-itest-cluster/ts-2/wal/instance: uuid: "e1881b44d0ad42d5b3c8cbe5e38aed5c" format_stamp: "Formatted at 2019-05-07 06:33:21 on dist-test-slave-dist-test-slave-rqp6" I0507 06:33:21.623411 14250 fs_manager.cc:503] Time spent creating directory manager: real 0.007s user 0.005s sys 0.004s I0507 06:33:21.624950 14250 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0507 06:33:21.625728 14250 file_cache.cc:466] Constructed file cache lbm with capacity 419430 I0507 06:33:21.629294 14250 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s I0507 06:33:21.629403 14250 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_2.1557210500371863-231/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_2.1557210500371863-231/raft_consensus-itest-cluster/ts-2/wal uuid: "e1881b44d0ad42d5b3c8cbe5e38aed5c" format_stamp: "Formatted at 2019-05-07 06:33:21 on dist-test-slave-dist-test-slave-rqp6" I0507 06:33:21.629644 14250 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_2.1557210500371863-231/raft_consensus-itest-cluster/ts-2/wal metadata directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_2.1557210500371863-231/raft_consensus-itest-cluster/ts-2/wal 1 data directories: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_2.1557210500371863-231/raft_consensus-itest-cluster/ts-2/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0507 06:33:21.670413 14250 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0507 06:33:21.672300 14250 ts_tablet_manager.cc:356] Loaded tablet metadata (0 total tablets, 0 live tablets) I0507 06:33:21.672384 14250 ts_tablet_manager.cc:376] Registered 0 tablets I0507 06:33:21.698900 14250 rpc_server.cc:205] RPC server started. Bound to: 127.0.57.195:41437 I0507 06:33:21.700269 14250 webserver.cc:308] Webserver started at http://127.0.57.195:34233/ using document root and password file I0507 06:33:21.701128 14250 server_base.cc:608] Dumped server information to /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_2.1557210500371863-231/raft_consensus-itest-cluster/ts-2/data/info.pb I0507 06:33:21.703012 231 external_mini_cluster.cc:995] Started /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver as pid 14250 I0507 06:33:21.721660 14369 heartbeater.cc:345] Connected to a master server at 127.0.57.254:44949 I0507 06:33:21.721896 14369 heartbeater.cc:426] Registering TS with master... I0507 06:33:21.722476 14369 heartbeater.cc:475] Master 127.0.57.254:44949 requested a full tablet report, sending... I0507 06:33:21.723592 13961 ts_manager.cc:159] Registered new tserver with Master: e1881b44d0ad42d5b3c8cbe5e38aed5c (127.0.57.195:41437) I0507 06:33:21.724069 231 external_mini_cluster.cc:623] 3 TS(s) registered with all masters I0507 06:33:21.724653 13961 master_service.cc:273] Signed X509 certificate for tserver {username='slave'} at 127.0.57.195:50951 I0507 06:33:21.735028 13961 catalog_manager.cc:1415] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50034: name: "TestTable" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 3 split_rows_range_bounds { } partition_schema { range_schema { columns { name: "key" } } } W0507 06:33:21.740229 13961 catalog_manager.cc:1574] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestTable in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers. I0507 06:33:21.764504 14306 tablet_service.cc:1016] Processing CreateTablet for tablet 6de7bc1405a248d39701963ce274a237 (table=TestTable [id=92f437ffe1294b9d88c53be9dff95585]), partition=RANGE (key) PARTITION UNBOUNDED I0507 06:33:21.765561 14306 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet 6de7bc1405a248d39701963ce274a237. 1 dirs total, 0 dirs full, 0 dirs failed I0507 06:33:21.776334 14061 tablet_service.cc:1016] Processing CreateTablet for tablet 6de7bc1405a248d39701963ce274a237 (table=TestTable [id=92f437ffe1294b9d88c53be9dff95585]), partition=RANGE (key) PARTITION UNBOUNDED I0507 06:33:21.776332 14184 tablet_service.cc:1016] Processing CreateTablet for tablet 6de7bc1405a248d39701963ce274a237 (table=TestTable [id=92f437ffe1294b9d88c53be9dff95585]), partition=RANGE (key) PARTITION UNBOUNDED I0507 06:33:21.777384 14184 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet 6de7bc1405a248d39701963ce274a237. 1 dirs total, 0 dirs full, 0 dirs failed I0507 06:33:21.777395 14061 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet 6de7bc1405a248d39701963ce274a237. 1 dirs total, 0 dirs full, 0 dirs failed I0507 06:33:21.783967 14388 tablet_bootstrap.cc:439] T 6de7bc1405a248d39701963ce274a237 P e1881b44d0ad42d5b3c8cbe5e38aed5c: Bootstrap starting. I0507 06:33:21.790712 14388 tablet_bootstrap.cc:590] T 6de7bc1405a248d39701963ce274a237 P e1881b44d0ad42d5b3c8cbe5e38aed5c: No blocks or log segments found. Creating new log. I0507 06:33:21.791857 14388 log.cc:525] T 6de7bc1405a248d39701963ce274a237 P e1881b44d0ad42d5b3c8cbe5e38aed5c: Log is configured to *not* fsync() on all Append() calls I0507 06:33:21.795276 14389 tablet_bootstrap.cc:439] T 6de7bc1405a248d39701963ce274a237 P 784329b2a1db4d4cb2e93c98aea55809: Bootstrap starting. I0507 06:33:21.795534 14391 tablet_bootstrap.cc:439] T 6de7bc1405a248d39701963ce274a237 P 539eecf299464551a0697eeab25b54c2: Bootstrap starting. I0507 06:33:21.798274 14389 tablet_bootstrap.cc:590] T 6de7bc1405a248d39701963ce274a237 P 784329b2a1db4d4cb2e93c98aea55809: No blocks or log segments found. Creating new log. I0507 06:33:21.798272 14391 tablet_bootstrap.cc:590] T 6de7bc1405a248d39701963ce274a237 P 539eecf299464551a0697eeab25b54c2: No blocks or log segments found. Creating new log. I0507 06:33:21.799433 14391 log.cc:525] T 6de7bc1405a248d39701963ce274a237 P 539eecf299464551a0697eeab25b54c2: Log is configured to *not* fsync() on all Append() calls I0507 06:33:21.799423 14389 log.cc:525] T 6de7bc1405a248d39701963ce274a237 P 784329b2a1db4d4cb2e93c98aea55809: Log is configured to *not* fsync() on all Append() calls I0507 06:33:21.801035 14388 tablet_bootstrap.cc:439] T 6de7bc1405a248d39701963ce274a237 P e1881b44d0ad42d5b3c8cbe5e38aed5c: No bootstrap required, opened a new log I0507 06:33:21.801383 14388 ts_tablet_manager.cc:1053] T 6de7bc1405a248d39701963ce274a237 P e1881b44d0ad42d5b3c8cbe5e38aed5c: Time spent bootstrapping tablet: real 0.018s user 0.003s sys 0.003s I0507 06:33:21.801638 14391 tablet_bootstrap.cc:439] T 6de7bc1405a248d39701963ce274a237 P 539eecf299464551a0697eeab25b54c2: No bootstrap required, opened a new log I0507 06:33:21.801892 14391 ts_tablet_manager.cc:1053] T 6de7bc1405a248d39701963ce274a237 P 539eecf299464551a0697eeab25b54c2: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.003s I0507 06:33:21.802214 14389 tablet_bootstrap.cc:439] T 6de7bc1405a248d39701963ce274a237 P 784329b2a1db4d4cb2e93c98aea55809: No bootstrap required, opened a new log I0507 06:33:21.802469 14389 ts_tablet_manager.cc:1053] T 6de7bc1405a248d39701963ce274a237 P 784329b2a1db4d4cb2e93c98aea55809: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.005s I0507 06:33:21.807947 14388 raft_consensus.cc:340] T 6de7bc1405a248d39701963ce274a237 P e1881b44d0ad42d5b3c8cbe5e38aed5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1881b44d0ad42d5b3c8cbe5e38aed5c" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 41437 } } peers { permanent_uuid: "539eecf299464551a0697eeab25b54c2" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 45599 } } peers { permanent_uuid: "784329b2a1db4d4cb2e93c98aea55809" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 46571 } } I0507 06:33:21.808349 14388 raft_consensus.cc:366] T 6de7bc1405a248d39701963ce274a237 P e1881b44d0ad42d5b3c8cbe5e38aed5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0507 06:33:21.808413 14388 raft_consensus.cc:705] T 6de7bc1405a248d39701963ce274a237 P e1881b44d0ad42d5b3c8cbe5e38aed5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1881b44d0ad42d5b3c8cbe5e38aed5c, State: Initialized, Role: FOLLOWER I0507 06:33:21.808950 14388 consensus_queue.cc:229] T 6de7bc1405a248d39701963ce274a237 P e1881b44d0ad42d5b3c8cbe5e38aed5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1881b44d0ad42d5b3c8cbe5e38aed5c" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 41437 } } peers { permanent_uuid: "539eecf299464551a0697eeab25b54c2" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 45599 } } peers { permanent_uuid: "784329b2a1db4d4cb2e93c98aea55809" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 46571 } } I0507 06:33:21.809861 14391 raft_consensus.cc:340] T 6de7bc1405a248d39701963ce274a237 P 539eecf299464551a0697eeab25b54c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1881b44d0ad42d5b3c8cbe5e38aed5c" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 41437 } } peers { permanent_uuid: "539eecf299464551a0697eeab25b54c2" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 45599 } } peers { permanent_uuid: "784329b2a1db4d4cb2e93c98aea55809" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 46571 } } I0507 06:33:21.810266 14391 raft_consensus.cc:366] T 6de7bc1405a248d39701963ce274a237 P 539eecf299464551a0697eeab25b54c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0507 06:33:21.810335 14391 raft_consensus.cc:705] T 6de7bc1405a248d39701963ce274a237 P 539eecf299464551a0697eeab25b54c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 539eecf299464551a0697eeab25b54c2, State: Initialized, Role: FOLLOWER I0507 06:33:21.810328 14389 raft_consensus.cc:340] T 6de7bc1405a248d39701963ce274a237 P 784329b2a1db4d4cb2e93c98aea55809 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1881b44d0ad42d5b3c8cbe5e38aed5c" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 41437 } } peers { permanent_uuid: "539eecf299464551a0697eeab25b54c2" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 45599 } } peers { permanent_uuid: "784329b2a1db4d4cb2e93c98aea55809" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 46571 } } I0507 06:33:21.810705 14389 raft_consensus.cc:366] T 6de7bc1405a248d39701963ce274a237 P 784329b2a1db4d4cb2e93c98aea55809 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0507 06:33:21.810776 14389 raft_consensus.cc:705] T 6de7bc1405a248d39701963ce274a237 P 784329b2a1db4d4cb2e93c98aea55809 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 784329b2a1db4d4cb2e93c98aea55809, State: Initialized, Role: FOLLOWER I0507 06:33:21.810783 14391 consensus_queue.cc:229] T 6de7bc1405a248d39701963ce274a237 P 539eecf299464551a0697eeab25b54c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1881b44d0ad42d5b3c8cbe5e38aed5c" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 41437 } } peers { permanent_uuid: "539eecf299464551a0697eeab25b54c2" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 45599 } } peers { permanent_uuid: "784329b2a1db4d4cb2e93c98aea55809" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 46571 } } I0507 06:33:21.811213 14389 consensus_queue.cc:229] T 6de7bc1405a248d39701963ce274a237 P 784329b2a1db4d4cb2e93c98aea55809 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1881b44d0ad42d5b3c8cbe5e38aed5c" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 41437 } } peers { permanent_uuid: "539eecf299464551a0697eeab25b54c2" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 45599 } } peers { permanent_uuid: "784329b2a1db4d4cb2e93c98aea55809" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 46571 } } I0507 06:33:21.811676 14369 heartbeater.cc:467] Master 127.0.57.254:44949 was elected leader, sending a full tablet report... I0507 06:33:21.812098 14124 heartbeater.cc:467] Master 127.0.57.254:44949 was elected leader, sending a full tablet report... I0507 06:33:21.812108 14247 heartbeater.cc:467] Master 127.0.57.254:44949 was elected leader, sending a full tablet report... I0507 06:33:21.812305 14388 ts_tablet_manager.cc:1081] T 6de7bc1405a248d39701963ce274a237 P e1881b44d0ad42d5b3c8cbe5e38aed5c: Time spent starting tablet: real 0.011s user 0.007s sys 0.004s I0507 06:33:21.813374 14391 ts_tablet_manager.cc:1081] T 6de7bc1405a248d39701963ce274a237 P 539eecf299464551a0697eeab25b54c2: Time spent starting tablet: real 0.011s user 0.009s sys 0.001s I0507 06:33:21.813374 14389 ts_tablet_manager.cc:1081] T 6de7bc1405a248d39701963ce274a237 P 784329b2a1db4d4cb2e93c98aea55809: Time spent starting tablet: real 0.011s user 0.008s sys 0.002s W0507 06:33:21.825071 14125 tablet.cc:1786] T 6de7bc1405a248d39701963ce274a237 P 784329b2a1db4d4cb2e93c98aea55809: Can't schedule compaction. Clean time has not been advanced past its initial value. W0507 06:33:21.903359 14370 tablet.cc:1786] T 6de7bc1405a248d39701963ce274a237 P e1881b44d0ad42d5b3c8cbe5e38aed5c: Can't schedule compaction. Clean time has not been advanced past its initial value. W0507 06:33:21.912612 14248 tablet.cc:1786] T 6de7bc1405a248d39701963ce274a237 P 539eecf299464551a0697eeab25b54c2: Can't schedule compaction. Clean time has not been advanced past its initial value. I0507 06:33:22.111176 14394 raft_consensus.cc:472] T 6de7bc1405a248d39701963ce274a237 P e1881b44d0ad42d5b3c8cbe5e38aed5c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0507 06:33:22.111456 14394 raft_consensus.cc:494] T 6de7bc1405a248d39701963ce274a237 P e1881b44d0ad42d5b3c8cbe5e38aed5c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1881b44d0ad42d5b3c8cbe5e38aed5c" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 41437 } } peers { permanent_uuid: "539eecf299464551a0697eeab25b54c2" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 45599 } } peers { permanent_uuid: "784329b2a1db4d4cb2e93c98aea55809" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 46571 } } I0507 06:33:22.113018 14394 leader_election.cc:296] T 6de7bc1405a248d39701963ce274a237 P e1881b44d0ad42d5b3c8cbe5e38aed5c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 539eecf299464551a0697eeab25b54c2 (127.0.57.194:45599), 784329b2a1db4d4cb2e93c98aea55809 (127.0.57.193:46571) I0507 06:33:22.119446 14081 tablet_service.cc:1286] Received RequestConsensusVote() RPC: tablet_id: "6de7bc1405a248d39701963ce274a237" candidate_uuid: "e1881b44d0ad42d5b3c8cbe5e38aed5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "784329b2a1db4d4cb2e93c98aea55809" is_pre_election: true I0507 06:33:22.119665 14204 tablet_service.cc:1286] Received RequestConsensusVote() RPC: tablet_id: "6de7bc1405a248d39701963ce274a237" candidate_uuid: "e1881b44d0ad42d5b3c8cbe5e38aed5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "539eecf299464551a0697eeab25b54c2" is_pre_election: true I0507 06:33:22.119949 14081 raft_consensus.cc:2346] T 6de7bc1405a248d39701963ce274a237 P 784329b2a1db4d4cb2e93c98aea55809 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e1881b44d0ad42d5b3c8cbe5e38aed5c in term 0. I0507 06:33:22.120033 14204 raft_consensus.cc:2346] T 6de7bc1405a248d39701963ce274a237 P 539eecf299464551a0697eeab25b54c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e1881b44d0ad42d5b3c8cbe5e38aed5c in term 0. I0507 06:33:22.120678 14257 leader_election.cc:310] T 6de7bc1405a248d39701963ce274a237 P e1881b44d0ad42d5b3c8cbe5e38aed5c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 784329b2a1db4d4cb2e93c98aea55809, e1881b44d0ad42d5b3c8cbe5e38aed5c; no voters: I0507 06:33:22.121187 14394 raft_consensus.cc:2640] T 6de7bc1405a248d39701963ce274a237 P e1881b44d0ad42d5b3c8cbe5e38aed5c [term 0 FOLLOWER]: Leader pre-election won for term 1 I0507 06:33:22.121294 14394 raft_consensus.cc:472] T 6de7bc1405a248d39701963ce274a237 P e1881b44d0ad42d5b3c8cbe5e38aed5c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0507 06:33:22.121364 14394 raft_consensus.cc:2886] T 6de7bc1405a248d39701963ce274a237 P e1881b44d0ad42d5b3c8cbe5e38aed5c [term 0 FOLLOWER]: Advancing to term 1 I0507 06:33:22.123438 14394 raft_consensus.cc:494] T 6de7bc1405a248d39701963ce274a237 P e1881b44d0ad42d5b3c8cbe5e38aed5c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1881b44d0ad42d5b3c8cbe5e38aed5c" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 41437 } } peers { permanent_uuid: "539eecf299464551a0697eeab25b54c2" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 45599 } } peers { permanent_uuid: "784329b2a1db4d4cb2e93c98aea55809" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 46571 } } I0507 06:33:22.124044 14394 leader_election.cc:296] T 6de7bc1405a248d39701963ce274a237 P e1881b44d0ad42d5b3c8cbe5e38aed5c [CANDIDATE]: Term 1 election: Requested vote from peers 539eecf299464551a0697eeab25b54c2 (127.0.57.194:45599), 784329b2a1db4d4cb2e93c98aea55809 (127.0.57.193:46571) I0507 06:33:22.124621 14204 tablet_service.cc:1286] Received RequestConsensusVote() RPC: tablet_id: "6de7bc1405a248d39701963ce274a237" candidate_uuid: "e1881b44d0ad42d5b3c8cbe5e38aed5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "539eecf299464551a0697eeab25b54c2" I0507 06:33:22.124630 14081 tablet_service.cc:1286] Received RequestConsensusVote() RPC: tablet_id: "6de7bc1405a248d39701963ce274a237" candidate_uuid: "e1881b44d0ad42d5b3c8cbe5e38aed5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "784329b2a1db4d4cb2e93c98aea55809" I0507 06:33:22.124801 14204 raft_consensus.cc:2886] T 6de7bc1405a248d39701963ce274a237 P 539eecf299464551a0697eeab25b54c2 [term 0 FOLLOWER]: Advancing to term 1 I0507 06:33:22.124809 14081 raft_consensus.cc:2886] T 6de7bc1405a248d39701963ce274a237 P 784329b2a1db4d4cb2e93c98aea55809 [term 0 FOLLOWER]: Advancing to term 1 I0507 06:33:22.126812 14204 raft_consensus.cc:2346] T 6de7bc1405a248d39701963ce274a237 P 539eecf299464551a0697eeab25b54c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e1881b44d0ad42d5b3c8cbe5e38aed5c in term 1. I0507 06:33:22.126889 14081 raft_consensus.cc:2346] T 6de7bc1405a248d39701963ce274a237 P 784329b2a1db4d4cb2e93c98aea55809 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e1881b44d0ad42d5b3c8cbe5e38aed5c in term 1. I0507 06:33:22.127341 14257 leader_election.cc:310] T 6de7bc1405a248d39701963ce274a237 P e1881b44d0ad42d5b3c8cbe5e38aed5c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 784329b2a1db4d4cb2e93c98aea55809, e1881b44d0ad42d5b3c8cbe5e38aed5c; no voters: I0507 06:33:22.127609 14394 raft_consensus.cc:2640] T 6de7bc1405a248d39701963ce274a237 P e1881b44d0ad42d5b3c8cbe5e38aed5c [term 1 FOLLOWER]: Leader election won for term 1 I0507 06:33:22.128046 14394 raft_consensus.cc:667] T 6de7bc1405a248d39701963ce274a237 P e1881b44d0ad42d5b3c8cbe5e38aed5c [term 1 LEADER]: Becoming Leader. State: Replica: e1881b44d0ad42d5b3c8cbe5e38aed5c, State: Running, Role: LEADER I0507 06:33:22.128491 14394 consensus_queue.cc:206] T 6de7bc1405a248d39701963ce274a237 P e1881b44d0ad42d5b3c8cbe5e38aed5c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1881b44d0ad42d5b3c8cbe5e38aed5c" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 41437 } } peers { permanent_uuid: "539eecf299464551a0697eeab25b54c2" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 45599 } } peers { permanent_uuid: "784329b2a1db4d4cb2e93c98aea55809" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 46571 } } I0507 06:33:22.132642 13959 catalog_manager.cc:3988] T 6de7bc1405a248d39701963ce274a237 P e1881b44d0ad42d5b3c8cbe5e38aed5c reported cstate change: term changed from 0 to 1, leader changed from to e1881b44d0ad42d5b3c8cbe5e38aed5c (127.0.57.195). New cstate: current_term: 1 leader_uuid: "e1881b44d0ad42d5b3c8cbe5e38aed5c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1881b44d0ad42d5b3c8cbe5e38aed5c" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 41437 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "539eecf299464551a0697eeab25b54c2" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 45599 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "784329b2a1db4d4cb2e93c98aea55809" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 46571 } health_report { overall_health: UNKNOWN } } } I0507 06:33:22.228037 231 external_mini_cluster.cc:623] 3 TS(s) registered with all masters I0507 06:33:22.229595 231 ts_itest-base.cc:232] Waiting for 1 tablets on tserver 784329b2a1db4d4cb2e93c98aea55809 to finish bootstrapping I0507 06:33:22.237128 231 ts_itest-base.cc:232] Waiting for 1 tablets on tserver 539eecf299464551a0697eeab25b54c2 to finish bootstrapping I0507 06:33:22.243532 231 ts_itest-base.cc:232] Waiting for 1 tablets on tserver e1881b44d0ad42d5b3c8cbe5e38aed5c to finish bootstrapping I0507 06:33:22.264652 231 cluster_itest_util.cc:258] Not converged past 1 yet: 0.0 0.0 1.1 I0507 06:33:22.368898 231 cluster_itest_util.cc:258] Not converged past 1 yet: 0.0 0.0 1.1 I0507 06:33:22.573297 231 cluster_itest_util.cc:258] Not converged past 1 yet: 0.0 0.0 1.1 I0507 06:33:22.684269 14204 raft_consensus.cc:1181] T 6de7bc1405a248d39701963ce274a237 P 539eecf299464551a0697eeab25b54c2 [term 1 FOLLOWER]: Refusing update from remote peer e1881b44d0ad42d5b3c8cbe5e38aed5c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch) I0507 06:33:22.685273 14402 consensus_queue.cc:984] T 6de7bc1405a248d39701963ce274a237 P e1881b44d0ad42d5b3c8cbe5e38aed5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "539eecf299464551a0697eeab25b54c2" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 45599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0507 06:33:22.695205 14081 raft_consensus.cc:1181] T 6de7bc1405a248d39701963ce274a237 P 784329b2a1db4d4cb2e93c98aea55809 [term 1 FOLLOWER]: Refusing update from remote peer e1881b44d0ad42d5b3c8cbe5e38aed5c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch) I0507 06:33:22.695950 14402 consensus_queue.cc:984] T 6de7bc1405a248d39701963ce274a237 P e1881b44d0ad42d5b3c8cbe5e38aed5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "784329b2a1db4d4cb2e93c98aea55809" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 46571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0507 06:33:22.877701 231 external_mini_cluster.cc:1172] Attempting to check leaks for /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver pid 13995 I0507 06:33:22.878768 14121 generic_service.cc:171] Checking for leaks (request via RPC) I0507 06:33:22.930177 231 external_mini_cluster.cc:1131] Killing /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver with pid 13995 I0507 06:33:22.945163 231 test_util.cc:212] Using random seed: 1690303600 I0507 06:33:22.956354 231 test_workload.cc:306] TestWorkload: Skipping table creation because table TestTable already exists I0507 06:33:22.962071 231 raft_consensus-itest-base.cc:271] Waiting until we've written at least 4MB... W0507 06:33:22.991226 14257 consensus_peers.cc:457] T 6de7bc1405a248d39701963ce274a237 P e1881b44d0ad42d5b3c8cbe5e38aed5c -> Peer 784329b2a1db4d4cb2e93c98aea55809 (127.0.57.193:46571): Couldn't send request to peer 784329b2a1db4d4cb2e93c98aea55809. Status: Network error: Client connection negotiation failed: client connection to 127.0.57.193:46571: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry. I0507 06:33:23.106544 231 raft_consensus-itest-base.cc:277] Waiting for log GC on e1881b44d0ad42d5b3c8cbe5e38aed5c I0507 06:33:23.108702 14370 maintenance_manager.cc:306] P e1881b44d0ad42d5b3c8cbe5e38aed5c: Scheduling FlushMRSOp(6de7bc1405a248d39701963ce274a237): 4601962 bytes log retention I0507 06:33:23.117413 14248 maintenance_manager.cc:306] P 539eecf299464551a0697eeab25b54c2: Scheduling FlushMRSOp(6de7bc1405a248d39701963ce274a237): 4207473 bytes log retention I0507 06:33:23.194794 14142 maintenance_manager.cc:501] P 539eecf299464551a0697eeab25b54c2: FlushMRSOp(6de7bc1405a248d39701963ce274a237) complete. Timing: real 0.077s user 0.060s sys 0.013s Metrics: {"bytes_written":136001,"cfile_init":1,"compiler_manager_pool.queue_time_us":7170,"data dirs.queue_time_us":5282,"data dirs.run_cpu_time_us":265,"data dirs.run_wall_time_us":2871,"drs_written":1,"lbm_read_time_us":71,"lbm_reads_lt_1ms":4,"lbm_write_time_us":588,"lbm_writes_lt_1ms":23,"rows_written":36,"thread_start_us":11857,"threads_started":2} I0507 06:33:23.196322 14248 maintenance_manager.cc:306] P 539eecf299464551a0697eeab25b54c2: Scheduling LogGCOp(6de7bc1405a248d39701963ce274a237): free 2629645 bytes of WAL I0507 06:33:23.196702 14142 log_reader.cc:377] T 6de7bc1405a248d39701963ce274a237: removed 3 log segments from log reader I0507 06:33:23.196835 14142 log.cc:863] T 6de7bc1405a248d39701963ce274a237 P 539eecf299464551a0697eeab25b54c2: Deleting log segment in path: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_2.1557210500371863-231/raft_consensus-itest-cluster/ts-1/wal/wals/6de7bc1405a248d39701963ce274a237/wal-000000001 (ops 1-8) I0507 06:33:23.196961 14142 log.cc:863] T 6de7bc1405a248d39701963ce274a237 P 539eecf299464551a0697eeab25b54c2: Deleting log segment in path: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_2.1557210500371863-231/raft_consensus-itest-cluster/ts-1/wal/wals/6de7bc1405a248d39701963ce274a237/wal-000000002 (ops 9-15) I0507 06:33:23.197048 14142 log.cc:863] T 6de7bc1405a248d39701963ce274a237 P 539eecf299464551a0697eeab25b54c2: Deleting log segment in path: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_2.1557210500371863-231/raft_consensus-itest-cluster/ts-1/wal/wals/6de7bc1405a248d39701963ce274a237/wal-000000003 (ops 16-21) I0507 06:33:23.197729 14142 maintenance_manager.cc:501] P 539eecf299464551a0697eeab25b54c2: LogGCOp(6de7bc1405a248d39701963ce274a237) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {} I0507 06:33:23.198009 14248 maintenance_manager.cc:306] P 539eecf299464551a0697eeab25b54c2: Scheduling UndoDeltaBlockGCOp(6de7bc1405a248d39701963ce274a237): 536 bytes on disk I0507 06:33:23.198707 14142 maintenance_manager.cc:501] P 539eecf299464551a0697eeab25b54c2: UndoDeltaBlockGCOp(6de7bc1405a248d39701963ce274a237) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":71,"lbm_reads_lt_1ms":4} I0507 06:33:23.220403 14264 maintenance_manager.cc:501] P e1881b44d0ad42d5b3c8cbe5e38aed5c: FlushMRSOp(6de7bc1405a248d39701963ce274a237) complete. Timing: real 0.111s user 0.091s sys 0.013s Metrics: {"bytes_written":136001,"cfile_init":1,"compiler_manager_pool.queue_time_us":510,"data dirs.queue_time_us":1683,"data dirs.run_cpu_time_us":208,"data dirs.run_wall_time_us":3766,"drs_written":1,"lbm_read_time_us":126,"lbm_reads_lt_1ms":4,"lbm_write_time_us":798,"lbm_writes_lt_1ms":23,"rows_written":36,"thread_start_us":1832,"threads_started":2} I0507 06:33:23.222028 14370 maintenance_manager.cc:306] P e1881b44d0ad42d5b3c8cbe5e38aed5c: Scheduling LogGCOp(6de7bc1405a248d39701963ce274a237): free 2761184 bytes of WAL I0507 06:33:23.222365 14264 log_reader.cc:377] T 6de7bc1405a248d39701963ce274a237: removed 3 log segments from log reader I0507 06:33:23.222440 14264 log.cc:863] T 6de7bc1405a248d39701963ce274a237 P e1881b44d0ad42d5b3c8cbe5e38aed5c: Deleting log segment in path: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_2.1557210500371863-231/raft_consensus-itest-cluster/ts-2/wal/wals/6de7bc1405a248d39701963ce274a237/wal-000000001 (ops 1-8) I0507 06:33:23.222517 14264 log.cc:863] T 6de7bc1405a248d39701963ce274a237 P e1881b44d0ad42d5b3c8cbe5e38aed5c: Deleting log segment in path: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_2.1557210500371863-231/raft_consensus-itest-cluster/ts-2/wal/wals/6de7bc1405a248d39701963ce274a237/wal-000000002 (ops 9-15) I0507 06:33:23.222605 14264 log.cc:863] T 6de7bc1405a248d39701963ce274a237 P e1881b44d0ad42d5b3c8cbe5e38aed5c: Deleting log segment in path: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_2.1557210500371863-231/raft_consensus-itest-cluster/ts-2/wal/wals/6de7bc1405a248d39701963ce274a237/wal-000000003 (ops 16-22) I0507 06:33:23.223284 14264 maintenance_manager.cc:501] P e1881b44d0ad42d5b3c8cbe5e38aed5c: LogGCOp(6de7bc1405a248d39701963ce274a237) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {} I0507 06:33:23.223588 14370 maintenance_manager.cc:306] P e1881b44d0ad42d5b3c8cbe5e38aed5c: Scheduling UndoDeltaBlockGCOp(6de7bc1405a248d39701963ce274a237): 536 bytes on disk I0507 06:33:23.224381 14264 maintenance_manager.cc:501] P e1881b44d0ad42d5b3c8cbe5e38aed5c: UndoDeltaBlockGCOp(6de7bc1405a248d39701963ce274a237) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":76,"lbm_reads_lt_1ms":4} I0507 06:33:23.230273 231 raft_consensus-itest-base.cc:282] Log GC complete on e1881b44d0ad42d5b3c8cbe5e38aed5c I0507 06:33:23.394817 14426 consensus_queue.cc:689] T 6de7bc1405a248d39701963ce274a237 P e1881b44d0ad42d5b3c8cbe5e38aed5c [LEADER]: The logs necessary to catch up peer 784329b2a1db4d4cb2e93c98aea55809 have been garbage collected. The follower will never be able to catch up (Not found: Failed to read ops 3..31: Segment 1 which contained index 3 has been GCed) I0507 06:33:23.394974 14426 consensus_queue.cc:544] T 6de7bc1405a248d39701963ce274a237 P e1881b44d0ad42d5b3c8cbe5e38aed5c [LEADER]: The logs necessary to catch up peer 784329b2a1db4d4cb2e93c98aea55809 have been garbage collected. The replica will never be able to catch up I0507 06:33:23.397496 13961 catalog_manager.cc:3739] Sending ChangeConfig:REMOVE_PEER on tablet 6de7bc1405a248d39701963ce274a237 (attempt 1) I0507 06:33:23.399937 14326 consensus_queue.cc:206] T 6de7bc1405a248d39701963ce274a237 P e1881b44d0ad42d5b3c8cbe5e38aed5c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 37, Committed index: 37, Last appended: 1.37, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 38 OBSOLETE_local: false peers { permanent_uuid: "e1881b44d0ad42d5b3c8cbe5e38aed5c" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 41437 } } peers { permanent_uuid: "539eecf299464551a0697eeab25b54c2" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 45599 } } I0507 06:33:23.402236 14204 raft_consensus.cc:1181] T 6de7bc1405a248d39701963ce274a237 P 539eecf299464551a0697eeab25b54c2 [term 1 FOLLOWER]: Refusing update from remote peer e1881b44d0ad42d5b3c8cbe5e38aed5c: Log matching property violated. Preceding OpId in replica: term: 1 index: 37. Preceding OpId from leader: term: 1 index: 38. (index mismatch) I0507 06:33:23.402897 14408 consensus_queue.cc:984] T 6de7bc1405a248d39701963ce274a237 P e1881b44d0ad42d5b3c8cbe5e38aed5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "539eecf299464551a0697eeab25b54c2" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 45599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 38, Last known committed idx: 37, Time since last communication: 0.000s I0507 06:33:23.405625 14402 raft_consensus.cc:2788] T 6de7bc1405a248d39701963ce274a237 P e1881b44d0ad42d5b3c8cbe5e38aed5c [term 1 LEADER]: Committing config change with OpId 1.38: config changed from index -1 to 38, VOTER 784329b2a1db4d4cb2e93c98aea55809 (127.0.57.193) evicted. New config: { opid_index: 38 OBSOLETE_local: false peers { permanent_uuid: "e1881b44d0ad42d5b3c8cbe5e38aed5c" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 41437 } } peers { permanent_uuid: "539eecf299464551a0697eeab25b54c2" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 45599 } } } I0507 06:33:23.406265 14204 raft_consensus.cc:2788] T 6de7bc1405a248d39701963ce274a237 P 539eecf299464551a0697eeab25b54c2 [term 1 FOLLOWER]: Committing config change with OpId 1.38: config changed from index -1 to 38, VOTER 784329b2a1db4d4cb2e93c98aea55809 (127.0.57.193) evicted. New config: { opid_index: 38 OBSOLETE_local: false peers { permanent_uuid: "e1881b44d0ad42d5b3c8cbe5e38aed5c" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 41437 } } peers { permanent_uuid: "539eecf299464551a0697eeab25b54c2" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 45599 } } } I0507 06:33:23.409354 13943 catalog_manager.cc:3531] ChangeConfig:REMOVE_PEER RPC for tablet 6de7bc1405a248d39701963ce274a237 with cas_config_opid_index -1: ChangeConfig:REMOVE_PEER succeeded (attempt 1) I0507 06:33:23.410359 13961 catalog_manager.cc:3988] T 6de7bc1405a248d39701963ce274a237 P e1881b44d0ad42d5b3c8cbe5e38aed5c reported cstate change: config changed from index -1 to 38, VOTER 784329b2a1db4d4cb2e93c98aea55809 (127.0.57.193) evicted. New cstate: current_term: 1 leader_uuid: "e1881b44d0ad42d5b3c8cbe5e38aed5c" committed_config { opid_index: 38 OBSOLETE_local: false peers { permanent_uuid: "e1881b44d0ad42d5b3c8cbe5e38aed5c" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 41437 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "539eecf299464551a0697eeab25b54c2" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 45599 } health_report { overall_health: HEALTHY } } } I0507 06:33:23.422003 13961 catalog_manager.cc:3390] Sending DeleteTablet(TABLET_DATA_TOMBSTONED) for tablet 6de7bc1405a248d39701963ce274a237 on 784329b2a1db4d4cb2e93c98aea55809 (127.0.57.193:46571) (TS 784329b2a1db4d4cb2e93c98aea55809 not found in new config with opid_index 38) W0507 06:33:23.423306 13942 catalog_manager.cc:3132] TS 784329b2a1db4d4cb2e93c98aea55809 (127.0.57.193:46571): Delete Tablet RPC failed for tablet 6de7bc1405a248d39701963ce274a237: Network error: Client connection negotiation failed: client connection to 127.0.57.193:46571: connect: Connection refused (error 111) I0507 06:33:23.423409 13942 catalog_manager.cc:3168] Scheduling retry of 6de7bc1405a248d39701963ce274a237 Delete Tablet RPC for TS=784329b2a1db4d4cb2e93c98aea55809 with a delay of 32 ms (attempt = 1) I0507 06:33:23.425539 14370 maintenance_manager.cc:306] P e1881b44d0ad42d5b3c8cbe5e38aed5c: Scheduling LogGCOp(6de7bc1405a248d39701963ce274a237): free 1840778 bytes of WAL I0507 06:33:23.425895 14264 log_reader.cc:377] T 6de7bc1405a248d39701963ce274a237: removed 2 log segments from log reader I0507 06:33:23.426048 14264 log.cc:863] T 6de7bc1405a248d39701963ce274a237 P e1881b44d0ad42d5b3c8cbe5e38aed5c: Deleting log segment in path: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_2.1557210500371863-231/raft_consensus-itest-cluster/ts-2/wal/wals/6de7bc1405a248d39701963ce274a237/wal-000000004 (ops 23-29) I0507 06:33:23.426131 14264 log.cc:863] T 6de7bc1405a248d39701963ce274a237 P e1881b44d0ad42d5b3c8cbe5e38aed5c: Deleting log segment in path: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_2.1557210500371863-231/raft_consensus-itest-cluster/ts-2/wal/wals/6de7bc1405a248d39701963ce274a237/wal-000000005 (ops 30-36) I0507 06:33:23.426615 14264 maintenance_manager.cc:501] P e1881b44d0ad42d5b3c8cbe5e38aed5c: LogGCOp(6de7bc1405a248d39701963ce274a237) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {} I0507 06:33:23.456044 13942 catalog_manager.cc:3390] Sending DeleteTablet(TABLET_DATA_TOMBSTONED) for tablet 6de7bc1405a248d39701963ce274a237 on 784329b2a1db4d4cb2e93c98aea55809 (127.0.57.193:46571) (TS 784329b2a1db4d4cb2e93c98aea55809 not found in new config with opid_index 38) W0507 06:33:23.457217 13942 catalog_manager.cc:3132] TS 784329b2a1db4d4cb2e93c98aea55809 (127.0.57.193:46571): Delete Tablet RPC failed for tablet 6de7bc1405a248d39701963ce274a237: Network error: Client connection negotiation failed: client connection to 127.0.57.193:46571: connect: Connection refused (error 111) I0507 06:33:23.457317 13942 catalog_manager.cc:3168] Scheduling retry of 6de7bc1405a248d39701963ce274a237 Delete Tablet RPC for TS=784329b2a1db4d4cb2e93c98aea55809 with a delay of 79 ms (attempt = 2) I0507 06:33:23.500337 14248 maintenance_manager.cc:306] P 539eecf299464551a0697eeab25b54c2: Scheduling LogGCOp(6de7bc1405a248d39701963ce274a237): free 1577828 bytes of WAL I0507 06:33:23.500797 14142 log_reader.cc:377] T 6de7bc1405a248d39701963ce274a237: removed 2 log segments from log reader I0507 06:33:23.500914 14142 log.cc:863] T 6de7bc1405a248d39701963ce274a237 P 539eecf299464551a0697eeab25b54c2: Deleting log segment in path: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_2.1557210500371863-231/raft_consensus-itest-cluster/ts-1/wal/wals/6de7bc1405a248d39701963ce274a237/wal-000000004 (ops 22-27) I0507 06:33:23.501042 14142 log.cc:863] T 6de7bc1405a248d39701963ce274a237 P 539eecf299464551a0697eeab25b54c2: Deleting log segment in path: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_2.1557210500371863-231/raft_consensus-itest-cluster/ts-1/wal/wals/6de7bc1405a248d39701963ce274a237/wal-000000005 (ops 28-33) I0507 06:33:23.501507 14142 maintenance_manager.cc:501] P 539eecf299464551a0697eeab25b54c2: LogGCOp(6de7bc1405a248d39701963ce274a237) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {} I0507 06:33:23.537180 13942 catalog_manager.cc:3390] Sending DeleteTablet(TABLET_DATA_TOMBSTONED) for tablet 6de7bc1405a248d39701963ce274a237 on 784329b2a1db4d4cb2e93c98aea55809 (127.0.57.193:46571) (TS 784329b2a1db4d4cb2e93c98aea55809 not found in new config with opid_index 38) W0507 06:33:23.538269 13942 catalog_manager.cc:3132] TS 784329b2a1db4d4cb2e93c98aea55809 (127.0.57.193:46571): Delete Tablet RPC failed for tablet 6de7bc1405a248d39701963ce274a237: Network error: Client connection negotiation failed: client connection to 127.0.57.193:46571: connect: Connection refused (error 111) I0507 06:33:23.538364 13942 catalog_manager.cc:3168] Scheduling retry of 6de7bc1405a248d39701963ce274a237 Delete Tablet RPC for TS=784329b2a1db4d4cb2e93c98aea55809 with a delay of 112 ms (attempt = 3) I0507 06:33:23.651154 13942 catalog_manager.cc:3390] Sending DeleteTablet(TABLET_DATA_TOMBSTONED) for tablet 6de7bc1405a248d39701963ce274a237 on 784329b2a1db4d4cb2e93c98aea55809 (127.0.57.193:46571) (TS 784329b2a1db4d4cb2e93c98aea55809 not found in new config with opid_index 38) W0507 06:33:23.652120 13942 catalog_manager.cc:3132] TS 784329b2a1db4d4cb2e93c98aea55809 (127.0.57.193:46571): Delete Tablet RPC failed for tablet 6de7bc1405a248d39701963ce274a237: Network error: Client connection negotiation failed: client connection to 127.0.57.193:46571: connect: Connection refused (error 111) I0507 06:33:23.652209 13942 catalog_manager.cc:3168] Scheduling retry of 6de7bc1405a248d39701963ce274a237 Delete Tablet RPC for TS=784329b2a1db4d4cb2e93c98aea55809 with a delay of 175 ms (attempt = 4) I0507 06:33:23.828341 13942 catalog_manager.cc:3390] Sending DeleteTablet(TABLET_DATA_TOMBSTONED) for tablet 6de7bc1405a248d39701963ce274a237 on 784329b2a1db4d4cb2e93c98aea55809 (127.0.57.193:46571) (TS 784329b2a1db4d4cb2e93c98aea55809 not found in new config with opid_index 38) W0507 06:33:23.829392 13942 catalog_manager.cc:3132] TS 784329b2a1db4d4cb2e93c98aea55809 (127.0.57.193:46571): Delete Tablet RPC failed for tablet 6de7bc1405a248d39701963ce274a237: Network error: Client connection negotiation failed: client connection to 127.0.57.193:46571: connect: Connection refused (error 111) I0507 06:33:23.829470 13942 catalog_manager.cc:3168] Scheduling retry of 6de7bc1405a248d39701963ce274a237 Delete Tablet RPC for TS=784329b2a1db4d4cb2e93c98aea55809 with a delay of 278 ms (attempt = 5) I0507 06:33:24.108920 13942 catalog_manager.cc:3390] Sending DeleteTablet(TABLET_DATA_TOMBSTONED) for tablet 6de7bc1405a248d39701963ce274a237 on 784329b2a1db4d4cb2e93c98aea55809 (127.0.57.193:46571) (TS 784329b2a1db4d4cb2e93c98aea55809 not found in new config with opid_index 38) W0507 06:33:24.109959 13942 catalog_manager.cc:3132] TS 784329b2a1db4d4cb2e93c98aea55809 (127.0.57.193:46571): Delete Tablet RPC failed for tablet 6de7bc1405a248d39701963ce274a237: Network error: Client connection negotiation failed: client connection to 127.0.57.193:46571: connect: Connection refused (error 111) I0507 06:33:24.110054 13942 catalog_manager.cc:3168] Scheduling retry of 6de7bc1405a248d39701963ce274a237 Delete Tablet RPC for TS=784329b2a1db4d4cb2e93c98aea55809 with a delay of 528 ms (attempt = 6) I0507 06:33:24.639385 13942 catalog_manager.cc:3390] Sending DeleteTablet(TABLET_DATA_TOMBSTONED) for tablet 6de7bc1405a248d39701963ce274a237 on 784329b2a1db4d4cb2e93c98aea55809 (127.0.57.193:46571) (TS 784329b2a1db4d4cb2e93c98aea55809 not found in new config with opid_index 38) W0507 06:33:24.640480 13942 catalog_manager.cc:3132] TS 784329b2a1db4d4cb2e93c98aea55809 (127.0.57.193:46571): Delete Tablet RPC failed for tablet 6de7bc1405a248d39701963ce274a237: Network error: Client connection negotiation failed: client connection to 127.0.57.193:46571: connect: Connection refused (error 111) I0507 06:33:24.640574 13942 catalog_manager.cc:3168] Scheduling retry of 6de7bc1405a248d39701963ce274a237 Delete Tablet RPC for TS=784329b2a1db4d4cb2e93c98aea55809 with a delay of 1060 ms (attempt = 7) I0507 06:33:25.232399 231 raft_consensus-itest-base.cc:298] Servers converged with original term 1 I0507 06:33:25.232475 231 raft_consensus-itest-base.cc:306] Restarting 784329b2a1db4d4cb2e93c98aea55809 I0507 06:33:25.233014 231 external_mini_cluster.cc:933] Running /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_2.1557210500371863-231/raft_consensus-itest-cluster/ts-0/wal --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_2.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data --block_manager=log --rpc_bind_addresses=127.0.57.193:46571 --local_ip_for_outbound_sockets=127.0.57.193 --webserver_port=35267 --webserver_interface=127.0.57.193 --tserver_master_addrs=127.0.57.254:44949 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_2.1557210500371863-231/raft_consensus-itest-cluster/ts-0/logs --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_2.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=true --follower_unavailable_considered_failed_sec=5 --log_compression_codec=none --log_cache_size_limit_mb=1 --log_segment_size_mb=1 --log_async_preallocate_segments=false --log_min_segments_to_retain=1 --log_max_segments_to_retain=3 --maintenance_manager_polling_interval_ms=100 --log_target_replay_size_mb=1 --max_cell_size_bytes=1000000 with env {} WARNING: Logging before InitGoogleLogging() is written to STDERR W0507 06:33:25.412228 14445 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0507 06:33:25.412451 14445 flags.cc:404] Enabled unsafe flag: --max_cell_size_bytes=1000000 W0507 06:33:25.412493 14445 flags.cc:404] Enabled unsafe flag: --never_fsync=true W0507 06:33:25.414659 14445 flags.cc:404] Enabled experimental flag: --log_compression_codec=none W0507 06:33:25.414719 14445 flags.cc:404] Enabled experimental flag: --log_max_segments_to_retain=3 W0507 06:33:25.414755 14445 flags.cc:404] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true W0507 06:33:25.414830 14445 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W0507 06:33:25.414922 14445 flags.cc:404] Enabled experimental flag: --log_target_replay_size_mb=1 W0507 06:33:25.414973 14445 flags.cc:404] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.57.193 I0507 06:33:25.419241 14445 tablet_server_main.cc:78] Tablet server non-default flags: --follower_unavailable_considered_failed_sec=5 --log_compression_codec=none --log_max_segments_to_retain=3 --log_cache_size_limit_mb=1 --log_async_preallocate_segments=false --log_segment_size_mb=1 --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_2.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_2.1557210500371863-231/raft_consensus-itest-cluster/ts-0/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.0.57.193:46571 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_2.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data/info.pb --webserver_interface=127.0.57.193 --webserver_port=35267 --max_cell_size_bytes=1000000 --tserver_master_addrs=127.0.57.254:44949 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --log_target_replay_size_mb=1 --maintenance_manager_polling_interval_ms=100 --enable_minidumps=false --local_ip_for_outbound_sockets=127.0.57.193 --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_2.1557210500371863-231/raft_consensus-itest-cluster/ts-0/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.10.0-cdh6.x-SNAPSHOT revision 7d0ee26697926e87af9ef395008799577f6ab43f build type FASTDEBUG built by jenkins at 06 May 2019 22:44:21 PST on quasar-knmpfx-1.vpc.cloudera.com build id 322017 ASAN enabled I0507 06:33:25.427664 14445 system_ntp.cc:202] NTP initialized. Skew: 500ppm Current error: 177016us I0507 06:33:25.428462 14445 fs_manager.cc:263] Metadata directory not provided I0507 06:33:25.428570 14445 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0507 06:33:25.434752 14445 fs_manager.cc:399] Time spent opening directory manager: real 0.005s user 0.001s sys 0.003s I0507 06:33:25.435524 14445 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0507 06:33:25.436156 14445 file_cache.cc:466] Constructed file cache lbm with capacity 419430 I0507 06:33:25.439662 14445 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s I0507 06:33:25.439774 14445 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_2.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_2.1557210500371863-231/raft_consensus-itest-cluster/ts-0/wal uuid: "784329b2a1db4d4cb2e93c98aea55809" format_stamp: "Formatted at 2019-05-07 06:33:20 on dist-test-slave-dist-test-slave-rqp6" I0507 06:33:25.441047 14445 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_2.1557210500371863-231/raft_consensus-itest-cluster/ts-0/wal metadata directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_2.1557210500371863-231/raft_consensus-itest-cluster/ts-0/wal 1 data directories: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_2.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0507 06:33:25.466409 14445 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0507 06:33:25.468205 14445 ts_tablet_manager.cc:344] Loading tablet metadata (0/1 complete) I0507 06:33:25.472301 14445 ts_tablet_manager.cc:356] Loaded tablet metadata (1 total tablets, 1 live tablets) I0507 06:33:25.472355 14445 ts_tablet_manager.cc:362] Registering tablets (0/1 complete) I0507 06:33:25.474869 14445 ts_tablet_manager.cc:376] Registered 1 tablets I0507 06:33:25.475307 14460 tablet_bootstrap.cc:439] T 6de7bc1405a248d39701963ce274a237 P 784329b2a1db4d4cb2e93c98aea55809: Bootstrap starting. I0507 06:33:25.483433 14460 log.cc:525] T 6de7bc1405a248d39701963ce274a237 P 784329b2a1db4d4cb2e93c98aea55809: Log is configured to *not* fsync() on all Append() calls I0507 06:33:25.489810 14460 tablet_bootstrap.cc:439] T 6de7bc1405a248d39701963ce274a237 P 784329b2a1db4d4cb2e93c98aea55809: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I0507 06:33:25.490362 14460 tablet_bootstrap.cc:439] T 6de7bc1405a248d39701963ce274a237 P 784329b2a1db4d4cb2e93c98aea55809: Bootstrap complete. I0507 06:33:25.490901 14460 ts_tablet_manager.cc:1053] T 6de7bc1405a248d39701963ce274a237 P 784329b2a1db4d4cb2e93c98aea55809: Time spent bootstrapping tablet: real 0.016s user 0.005s sys 0.007s I0507 06:33:25.505425 14460 raft_consensus.cc:340] T 6de7bc1405a248d39701963ce274a237 P 784329b2a1db4d4cb2e93c98aea55809 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1881b44d0ad42d5b3c8cbe5e38aed5c" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 41437 } } peers { permanent_uuid: "539eecf299464551a0697eeab25b54c2" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 45599 } } peers { permanent_uuid: "784329b2a1db4d4cb2e93c98aea55809" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 46571 } } I0507 06:33:25.508774 14460 raft_consensus.cc:705] T 6de7bc1405a248d39701963ce274a237 P 784329b2a1db4d4cb2e93c98aea55809 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 784329b2a1db4d4cb2e93c98aea55809, State: Initialized, Role: FOLLOWER I0507 06:33:25.510596 14460 consensus_queue.cc:229] T 6de7bc1405a248d39701963ce274a237 P 784329b2a1db4d4cb2e93c98aea55809 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1881b44d0ad42d5b3c8cbe5e38aed5c" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 41437 } } peers { permanent_uuid: "539eecf299464551a0697eeab25b54c2" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 45599 } } peers { permanent_uuid: "784329b2a1db4d4cb2e93c98aea55809" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 46571 } } I0507 06:33:25.512614 14460 ts_tablet_manager.cc:1081] T 6de7bc1405a248d39701963ce274a237 P 784329b2a1db4d4cb2e93c98aea55809: Time spent starting tablet: real 0.022s user 0.008s sys 0.013s I0507 06:33:25.527648 14445 rpc_server.cc:205] RPC server started. Bound to: 127.0.57.193:46571 I0507 06:33:25.528834 14445 webserver.cc:308] Webserver started at http://127.0.57.193:35267/ using document root and password file I0507 06:33:25.529510 14445 server_base.cc:608] Dumped server information to /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_2.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data/info.pb I0507 06:33:25.531678 231 external_mini_cluster.cc:995] Started /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver as pid 14445 I0507 06:33:25.539944 14567 heartbeater.cc:345] Connected to a master server at 127.0.57.254:44949 I0507 06:33:25.540161 14567 heartbeater.cc:426] Registering TS with master... I0507 06:33:25.540339 231 external_mini_cluster.cc:1172] Attempting to check leaks for /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-master pid 13935 I0507 06:33:25.540714 14567 heartbeater.cc:475] Master 127.0.57.254:44949 requested a full tablet report, sending... I0507 06:33:25.541031 13991 generic_service.cc:171] Checking for leaks (request via RPC) I0507 06:33:25.542332 13961 ts_manager.cc:159] Re-registered known tserver with Master: 784329b2a1db4d4cb2e93c98aea55809 (127.0.57.193:46571) I0507 06:33:25.624619 13961 catalog_manager.cc:3390] Sending DeleteTablet(TABLET_DATA_TOMBSTONED) for tablet 6de7bc1405a248d39701963ce274a237 on 784329b2a1db4d4cb2e93c98aea55809 (127.0.57.193:46571) (Replica has no consensus available (current committed config index is 38)) I0507 06:33:25.624868 231 external_mini_cluster.cc:1131] Killing /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-master with pid 13935 W0507 06:33:25.647060 14567 heartbeater.cc:587] Failed to heartbeat to 127.0.57.254:44949 (0 consecutive failures): Network error: Failed to send heartbeat to master: recv got EOF from 127.0.57.254:44949 (error 108) W0507 06:33:25.647753 14572 negotiation.cc:313] Failed RPC negotiation. Trace: 0507 06:33:25.625614 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.0.0.1:33456 0507 06:33:25.633348 (+ 7734us) server_negotiation.cc:184] Beginning negotiation 0507 06:33:25.633357 (+ 9us) server_negotiation.cc:373] Waiting for connection header 0507 06:33:25.646364 (+ 13007us) negotiation.cc:304] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33456: BlockingRecv error: recv got EOF from 127.0.0.1:33456 (error 108) Metrics: {"server-negotiator.queue_time_us":7472,"thread_start_us":7696,"threads_started":1} I0507 06:33:25.647753 231 external_mini_cluster.cc:933] Running /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-master /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-master --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_2.1557210500371863-231/raft_consensus-itest-cluster/master-0/wal --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_2.1557210500371863-231/raft_consensus-itest-cluster/master-0/data --block_manager=log --webserver_interface=localhost --ipki_ca_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.0.57.254:44949 --webserver_interface=127.0.57.254 --webserver_port=33821 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_2.1557210500371863-231/raft_consensus-itest-cluster/master-0/logs --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_2.1557210500371863-231/raft_consensus-itest-cluster/master-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=true --master_add_server_when_underreplicated=false --rpc_reuseport=true --master_add_server_when_underreplicated=true with env {} W0507 06:33:25.649324 14567 heartbeater.cc:380] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts. WARNING: Logging before InitGoogleLogging() is written to STDERR W0507 06:33:25.868434 14573 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0507 06:33:25.868803 14573 flags.cc:404] Enabled unsafe flag: --never_fsync=true W0507 06:33:25.871459 14573 flags.cc:404] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true W0507 06:33:25.871542 14573 flags.cc:404] Enabled experimental flag: --ipki_ca_key_size=1024 W0507 06:33:25.871588 14573 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W0507 06:33:25.871624 14573 flags.cc:404] Enabled experimental flag: --tsk_num_rsa_bits=512 W0507 06:33:25.871675 14573 flags.cc:404] Enabled experimental flag: --rpc_reuseport=true I0507 06:33:25.876734 14573 master_main.cc:73] Master server non-default flags: --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_2.1557210500371863-231/raft_consensus-itest-cluster/master-0/data --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_2.1557210500371863-231/raft_consensus-itest-cluster/master-0/wal --ipki_ca_key_size=1024 --ipki_server_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.0.57.254:44949 --rpc_reuseport=true --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_2.1557210500371863-231/raft_consensus-itest-cluster/master-0/data/info.pb --webserver_interface=127.0.57.254 --webserver_port=33821 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_2.1557210500371863-231/raft_consensus-itest-cluster/master-0/logs --logbuflevel=-1 --logtostderr=true Master server version: kudu 1.10.0-cdh6.x-SNAPSHOT revision 7d0ee26697926e87af9ef395008799577f6ab43f build type FASTDEBUG built by jenkins at 06 May 2019 22:44:21 PST on quasar-knmpfx-1.vpc.cloudera.com build id 322017 ASAN enabled I0507 06:33:25.884063 14573 system_ntp.cc:202] NTP initialized. Skew: 500ppm Current error: 177016us I0507 06:33:25.885597 14573 fs_manager.cc:263] Metadata directory not provided I0507 06:33:25.885686 14573 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0507 06:33:25.893048 14573 fs_manager.cc:399] Time spent opening directory manager: real 0.005s user 0.003s sys 0.005s I0507 06:33:25.893765 14573 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0507 06:33:25.894387 14573 file_cache.cc:466] Constructed file cache lbm with capacity 419430 I0507 06:33:25.897758 14573 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s I0507 06:33:25.897855 14573 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_2.1557210500371863-231/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_2.1557210500371863-231/raft_consensus-itest-cluster/master-0/wal uuid: "16571e2dbc9d46159c1792ce321cd0d7" format_stamp: "Formatted at 2019-05-07 06:33:20 on dist-test-slave-dist-test-slave-rqp6" I0507 06:33:25.898905 14573 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_2.1557210500371863-231/raft_consensus-itest-cluster/master-0/wal metadata directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_2.1557210500371863-231/raft_consensus-itest-cluster/master-0/wal 1 data directories: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_2.1557210500371863-231/raft_consensus-itest-cluster/master-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0507 06:33:25.924875 14573 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0507 06:33:25.940131 14573 rpc_server.cc:205] RPC server started. Bound to: 127.0.57.254:44949 I0507 06:33:25.941428 14573 webserver.cc:308] Webserver started at http://127.0.57.254:33821/ using document root and password file I0507 06:33:25.942425 14573 server_base.cc:608] Dumped server information to /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_2.1557210500371863-231/raft_consensus-itest-cluster/master-0/data/info.pb I0507 06:33:25.951041 231 external_mini_cluster.cc:995] Started /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-master as pid 14573 I0507 06:33:25.951840 14632 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 16571e2dbc9d46159c1792ce321cd0d7: Bootstrap starting. I0507 06:33:25.973853 14632 log.cc:525] T 00000000000000000000000000000000 P 16571e2dbc9d46159c1792ce321cd0d7: Log is configured to *not* fsync() on all Append() calls I0507 06:33:25.983620 14632 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 16571e2dbc9d46159c1792ce321cd0d7: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=4 ignored=0} mutations{seen=3 ignored=0} orphaned_commits=0. Pending: 0 replicates I0507 06:33:25.984105 14632 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 16571e2dbc9d46159c1792ce321cd0d7: Bootstrap complete. I0507 06:33:25.990774 14632 raft_consensus.cc:340] T 00000000000000000000000000000000 P 16571e2dbc9d46159c1792ce321cd0d7 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16571e2dbc9d46159c1792ce321cd0d7" member_type: VOTER } I0507 06:33:25.991505 14632 raft_consensus.cc:705] T 00000000000000000000000000000000 P 16571e2dbc9d46159c1792ce321cd0d7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16571e2dbc9d46159c1792ce321cd0d7, State: Initialized, Role: FOLLOWER I0507 06:33:25.992036 14632 consensus_queue.cc:229] T 00000000000000000000000000000000 P 16571e2dbc9d46159c1792ce321cd0d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16571e2dbc9d46159c1792ce321cd0d7" member_type: VOTER } I0507 06:33:25.992218 14632 raft_consensus.cc:380] T 00000000000000000000000000000000 P 16571e2dbc9d46159c1792ce321cd0d7 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0507 06:33:25.992308 14632 raft_consensus.cc:472] T 00000000000000000000000000000000 P 16571e2dbc9d46159c1792ce321cd0d7 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0507 06:33:25.992395 14632 raft_consensus.cc:2886] T 00000000000000000000000000000000 P 16571e2dbc9d46159c1792ce321cd0d7 [term 1 FOLLOWER]: Advancing to term 2 I0507 06:33:25.996445 14632 raft_consensus.cc:494] T 00000000000000000000000000000000 P 16571e2dbc9d46159c1792ce321cd0d7 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16571e2dbc9d46159c1792ce321cd0d7" member_type: VOTER } I0507 06:33:25.996937 14632 leader_election.cc:310] T 00000000000000000000000000000000 P 16571e2dbc9d46159c1792ce321cd0d7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 16571e2dbc9d46159c1792ce321cd0d7; no voters: I0507 06:33:25.997706 14636 raft_consensus.cc:2640] T 00000000000000000000000000000000 P 16571e2dbc9d46159c1792ce321cd0d7 [term 2 FOLLOWER]: Leader election won for term 2 I0507 06:33:25.998023 14632 leader_election.cc:296] T 00000000000000000000000000000000 P 16571e2dbc9d46159c1792ce321cd0d7 [CANDIDATE]: Term 2 election: Requested vote from peers I0507 06:33:25.998811 14636 raft_consensus.cc:667] T 00000000000000000000000000000000 P 16571e2dbc9d46159c1792ce321cd0d7 [term 2 LEADER]: Becoming Leader. State: Replica: 16571e2dbc9d46159c1792ce321cd0d7, State: Running, Role: LEADER I0507 06:33:25.999198 14636 consensus_queue.cc:206] T 00000000000000000000000000000000 P 16571e2dbc9d46159c1792ce321cd0d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16571e2dbc9d46159c1792ce321cd0d7" member_type: VOTER } I0507 06:33:26.002327 14637 sys_catalog.cc:337] T 00000000000000000000000000000000 P 16571e2dbc9d46159c1792ce321cd0d7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "16571e2dbc9d46159c1792ce321cd0d7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16571e2dbc9d46159c1792ce321cd0d7" member_type: VOTER } } I0507 06:33:26.002899 14637 sys_catalog.cc:340] T 00000000000000000000000000000000 P 16571e2dbc9d46159c1792ce321cd0d7 [sys.catalog]: This master's current role is: LEADER I0507 06:33:26.003057 14638 sys_catalog.cc:337] T 00000000000000000000000000000000 P 16571e2dbc9d46159c1792ce321cd0d7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 16571e2dbc9d46159c1792ce321cd0d7. Latest consensus state: current_term: 2 leader_uuid: "16571e2dbc9d46159c1792ce321cd0d7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16571e2dbc9d46159c1792ce321cd0d7" member_type: VOTER } } I0507 06:33:26.003393 14638 sys_catalog.cc:340] T 00000000000000000000000000000000 P 16571e2dbc9d46159c1792ce321cd0d7 [sys.catalog]: This master's current role is: LEADER I0507 06:33:26.003428 14632 sys_catalog.cc:337] T 00000000000000000000000000000000 P 16571e2dbc9d46159c1792ce321cd0d7 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 2 leader_uuid: "16571e2dbc9d46159c1792ce321cd0d7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16571e2dbc9d46159c1792ce321cd0d7" member_type: VOTER } } I0507 06:33:26.003543 14632 sys_catalog.cc:340] T 00000000000000000000000000000000 P 16571e2dbc9d46159c1792ce321cd0d7 [sys.catalog]: This master's current role is: LEADER I0507 06:33:26.004720 14632 sys_catalog.cc:424] T 00000000000000000000000000000000 P 16571e2dbc9d46159c1792ce321cd0d7 [sys.catalog]: configured and running, proceeding with master startup. I0507 06:33:26.005249 14642 catalog_manager.cc:1055] Loading table and tablet metadata into memory... I0507 06:33:26.006837 14643 catalog_manager.cc:538] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting I0507 06:33:26.010862 14642 catalog_manager.cc:356] Loaded metadata for table TestTable [id=92f437ffe1294b9d88c53be9dff95585] I0507 06:33:26.012307 14642 catalog_manager.cc:410] Loaded metadata for tablet 6de7bc1405a248d39701963ce274a237 (table TestTable [id=92f437ffe1294b9d88c53be9dff95585]) I0507 06:33:26.012476 14642 catalog_manager.cc:1066] Initializing Kudu internal certificate authority... I0507 06:33:26.019145 14642 catalog_manager.cc:1075] Loading token signing keys... I0507 06:33:26.020078 14642 catalog_manager.cc:4303] T 00000000000000000000000000000000 P 16571e2dbc9d46159c1792ce321cd0d7: Loaded TSK: 0 I0507 06:33:26.437278 14599 master_service.cc:229] Got heartbeat from unknown tserver (permanent_uuid: "e1881b44d0ad42d5b3c8cbe5e38aed5c" instance_seqno: 1557210801693415) as {username='slave'} at 127.0.57.195:46809; Asking this server to re-register. I0507 06:33:26.437278 14598 master_service.cc:229] Got heartbeat from unknown tserver (permanent_uuid: "539eecf299464551a0697eeab25b54c2" instance_seqno: 1557210801401517) as {username='slave'} at 127.0.57.194:55661; Asking this server to re-register. I0507 06:33:26.438189 14247 heartbeater.cc:426] Registering TS with master... I0507 06:33:26.438194 14369 heartbeater.cc:426] Registering TS with master... I0507 06:33:26.438413 14247 heartbeater.cc:475] Master 127.0.57.254:44949 requested a full tablet report, sending... I0507 06:33:26.438413 14369 heartbeater.cc:475] Master 127.0.57.254:44949 requested a full tablet report, sending... I0507 06:33:26.439862 14598 ts_manager.cc:159] Registered new tserver with Master: 539eecf299464551a0697eeab25b54c2 (127.0.57.194:45599) I0507 06:33:26.439982 14599 ts_manager.cc:159] Registered new tserver with Master: e1881b44d0ad42d5b3c8cbe5e38aed5c (127.0.57.195:41437) I0507 06:33:26.442137 14599 catalog_manager.cc:3616] Sending ChangeConfig:ADD_PEER:NON_VOTER on tablet 6de7bc1405a248d39701963ce274a237 (attempt 1) W0507 06:33:26.442730 14599 catalog_manager.cc:3675] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 6de7bc1405a248d39701963ce274a237 with cas_config_opid_index 38: no extra replica candidate found for tablet 6de7bc1405a248d39701963ce274a237 (table TestTable [id=92f437ffe1294b9d88c53be9dff95585]): Not found: could not select location for extra replica: not enough tablet servers to satisfy replica placement policy: the total number of registered tablet servers (2) does not allow for adding an extra replica; consider bringing up more to have at least 4 tablet servers up and running I0507 06:33:26.442817 14599 catalog_manager.cc:3168] Scheduling retry of ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 6de7bc1405a248d39701963ce274a237 with cas_config_opid_index 38 with a delay of 35 ms (attempt = 1) I0507 06:33:26.478943 14582 catalog_manager.cc:3616] Sending ChangeConfig:ADD_PEER:NON_VOTER on tablet 6de7bc1405a248d39701963ce274a237 (attempt 2) I0507 06:33:26.479296 14582 catalog_manager.cc:3168] Scheduling retry of ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 6de7bc1405a248d39701963ce274a237 with cas_config_opid_index 38 with a delay of 33 ms (attempt = 2) I0507 06:33:26.513239 14582 catalog_manager.cc:3616] Sending ChangeConfig:ADD_PEER:NON_VOTER on tablet 6de7bc1405a248d39701963ce274a237 (attempt 3) I0507 06:33:26.513659 14582 catalog_manager.cc:3168] Scheduling retry of ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 6de7bc1405a248d39701963ce274a237 with cas_config_opid_index 38 with a delay of 90 ms (attempt = 3) I0507 06:33:26.604743 14582 catalog_manager.cc:3616] Sending ChangeConfig:ADD_PEER:NON_VOTER on tablet 6de7bc1405a248d39701963ce274a237 (attempt 4) I0507 06:33:26.605048 14582 catalog_manager.cc:3168] Scheduling retry of ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 6de7bc1405a248d39701963ce274a237 with cas_config_opid_index 38 with a delay of 161 ms (attempt = 4) I0507 06:33:26.655196 14567 heartbeater.cc:345] Connected to a master server at 127.0.57.254:44949 I0507 06:33:26.655321 14567 heartbeater.cc:426] Registering TS with master... I0507 06:33:26.655534 14567 heartbeater.cc:475] Master 127.0.57.254:44949 requested a full tablet report, sending... I0507 06:33:26.656623 14599 ts_manager.cc:159] Registered new tserver with Master: 784329b2a1db4d4cb2e93c98aea55809 (127.0.57.193:46571) I0507 06:33:26.657380 14599 catalog_manager.cc:3390] Sending DeleteTablet(TABLET_DATA_TOMBSTONED) for tablet 6de7bc1405a248d39701963ce274a237 on 784329b2a1db4d4cb2e93c98aea55809 (127.0.57.193:46571) (Replica has no consensus available (current committed config index is 38)) I0507 06:33:26.660117 14599 master_service.cc:273] Signed X509 certificate for tserver {username='slave'} at 127.0.57.193:35395 I0507 06:33:26.670104 14504 tablet_service.cc:1057] Processing DeleteTablet for tablet 6de7bc1405a248d39701963ce274a237 with delete_type TABLET_DATA_TOMBSTONED (Replica has no consensus available (current committed config index is 38)) from {username='slave'} at 127.0.0.1:33460 I0507 06:33:26.672788 14651 tablet_replica.cc:264] T 6de7bc1405a248d39701963ce274a237 P 784329b2a1db4d4cb2e93c98aea55809: stopping tablet replica I0507 06:33:26.673130 14651 raft_consensus.cc:2143] T 6de7bc1405a248d39701963ce274a237 P 784329b2a1db4d4cb2e93c98aea55809 [term 1 FOLLOWER]: Raft consensus shutting down. I0507 06:33:26.673485 14651 raft_consensus.cc:2170] T 6de7bc1405a248d39701963ce274a237 P 784329b2a1db4d4cb2e93c98aea55809 [term 1 FOLLOWER]: Raft consensus is shut down! I0507 06:33:26.674723 14651 ts_tablet_manager.cc:1377] T 6de7bc1405a248d39701963ce274a237 P 784329b2a1db4d4cb2e93c98aea55809: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED I0507 06:33:26.680409 14651 ts_tablet_manager.cc:1390] T 6de7bc1405a248d39701963ce274a237 P 784329b2a1db4d4cb2e93c98aea55809: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.1 I0507 06:33:26.680521 14651 log.cc:990] T 6de7bc1405a248d39701963ce274a237 P 784329b2a1db4d4cb2e93c98aea55809: Deleting WAL directory at /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_2.1557210500371863-231/raft_consensus-itest-cluster/ts-0/wal/wals/6de7bc1405a248d39701963ce274a237 I0507 06:33:26.682018 14580 catalog_manager.cc:3368] TS 784329b2a1db4d4cb2e93c98aea55809 (127.0.57.193:46571): tablet 6de7bc1405a248d39701963ce274a237 (table TestTable [id=92f437ffe1294b9d88c53be9dff95585]) successfully deleted I0507 06:33:26.766433 14582 catalog_manager.cc:3616] Sending ChangeConfig:ADD_PEER:NON_VOTER on tablet 6de7bc1405a248d39701963ce274a237 (attempt 5) I0507 06:33:26.773133 14326 consensus_queue.cc:206] T 6de7bc1405a248d39701963ce274a237 P e1881b44d0ad42d5b3c8cbe5e38aed5c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 38, Committed index: 38, Last appended: 1.38, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 39 OBSOLETE_local: false peers { permanent_uuid: "e1881b44d0ad42d5b3c8cbe5e38aed5c" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 41437 } } peers { permanent_uuid: "539eecf299464551a0697eeab25b54c2" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 45599 } } peers { permanent_uuid: "784329b2a1db4d4cb2e93c98aea55809" member_type: NON_VOTER last_known_addr { host: "127.0.57.193" port: 46571 } attrs { promote: true } } I0507 06:33:26.775956 14204 raft_consensus.cc:1181] T 6de7bc1405a248d39701963ce274a237 P 539eecf299464551a0697eeab25b54c2 [term 1 FOLLOWER]: Refusing update from remote peer e1881b44d0ad42d5b3c8cbe5e38aed5c: Log matching property violated. Preceding OpId in replica: term: 1 index: 38. Preceding OpId from leader: term: 1 index: 39. (index mismatch) I0507 06:33:26.776471 14408 consensus_queue.cc:984] T 6de7bc1405a248d39701963ce274a237 P e1881b44d0ad42d5b3c8cbe5e38aed5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "539eecf299464551a0697eeab25b54c2" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 45599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 39, Last known committed idx: 38, Time since last communication: 0.000s I0507 06:33:26.779898 14654 raft_consensus.cc:2788] T 6de7bc1405a248d39701963ce274a237 P e1881b44d0ad42d5b3c8cbe5e38aed5c [term 1 LEADER]: Committing config change with OpId 1.39: config changed from index 38 to 39, NON_VOTER 784329b2a1db4d4cb2e93c98aea55809 (127.0.57.193) added. New config: { opid_index: 39 OBSOLETE_local: false peers { permanent_uuid: "e1881b44d0ad42d5b3c8cbe5e38aed5c" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 41437 } } peers { permanent_uuid: "539eecf299464551a0697eeab25b54c2" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 45599 } } peers { permanent_uuid: "784329b2a1db4d4cb2e93c98aea55809" member_type: NON_VOTER last_known_addr { host: "127.0.57.193" port: 46571 } attrs { promote: true } } } W0507 06:33:26.780568 14257 consensus_peers.cc:457] T 6de7bc1405a248d39701963ce274a237 P e1881b44d0ad42d5b3c8cbe5e38aed5c -> Peer 784329b2a1db4d4cb2e93c98aea55809 (127.0.57.193:46571): Couldn't send request to peer 784329b2a1db4d4cb2e93c98aea55809. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry. I0507 06:33:26.780380 14204 raft_consensus.cc:2788] T 6de7bc1405a248d39701963ce274a237 P 539eecf299464551a0697eeab25b54c2 [term 1 FOLLOWER]: Committing config change with OpId 1.39: config changed from index 38 to 39, NON_VOTER 784329b2a1db4d4cb2e93c98aea55809 (127.0.57.193) added. New config: { opid_index: 39 OBSOLETE_local: false peers { permanent_uuid: "e1881b44d0ad42d5b3c8cbe5e38aed5c" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 41437 } } peers { permanent_uuid: "539eecf299464551a0697eeab25b54c2" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 45599 } } peers { permanent_uuid: "784329b2a1db4d4cb2e93c98aea55809" member_type: NON_VOTER last_known_addr { host: "127.0.57.193" port: 46571 } attrs { promote: true } } } I0507 06:33:26.783020 14581 catalog_manager.cc:3531] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 6de7bc1405a248d39701963ce274a237 with cas_config_opid_index 38: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 5) I0507 06:33:26.784631 14598 catalog_manager.cc:3988] T 6de7bc1405a248d39701963ce274a237 P 539eecf299464551a0697eeab25b54c2 reported cstate change: config changed from index 38 to 39, NON_VOTER 784329b2a1db4d4cb2e93c98aea55809 (127.0.57.193) added. New cstate: current_term: 1 leader_uuid: "e1881b44d0ad42d5b3c8cbe5e38aed5c" committed_config { opid_index: 39 OBSOLETE_local: false peers { permanent_uuid: "e1881b44d0ad42d5b3c8cbe5e38aed5c" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 41437 } } peers { permanent_uuid: "539eecf299464551a0697eeab25b54c2" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 45599 } } peers { permanent_uuid: "784329b2a1db4d4cb2e93c98aea55809" member_type: NON_VOTER last_known_addr { host: "127.0.57.193" port: 46571 } attrs { promote: true } } } I0507 06:33:27.273383 14660 ts_tablet_manager.cc:676] T 6de7bc1405a248d39701963ce274a237 P 784329b2a1db4d4cb2e93c98aea55809: Initiating tablet copy from peer e1881b44d0ad42d5b3c8cbe5e38aed5c (127.0.57.195:41437) I0507 06:33:27.273808 14660 tablet_copy_client.cc:240] T 6de7bc1405a248d39701963ce274a237 P 784329b2a1db4d4cb2e93c98aea55809: tablet copy: Beginning tablet copy session from remote peer at address 127.0.57.195:41437 I0507 06:33:27.280391 14346 tablet_copy_service.cc:135] P e1881b44d0ad42d5b3c8cbe5e38aed5c: Received BeginTabletCopySession request for tablet 6de7bc1405a248d39701963ce274a237 from peer 784329b2a1db4d4cb2e93c98aea55809 ({username='slave'} at 127.0.57.193:50543) I0507 06:33:27.280612 14346 tablet_copy_service.cc:156] P e1881b44d0ad42d5b3c8cbe5e38aed5c: Beginning new tablet copy session on tablet 6de7bc1405a248d39701963ce274a237 from peer 784329b2a1db4d4cb2e93c98aea55809 at {username='slave'} at 127.0.57.193:50543: session id = 784329b2a1db4d4cb2e93c98aea55809-6de7bc1405a248d39701963ce274a237 I0507 06:33:27.283362 14346 tablet_copy_source_session.cc:202] T 6de7bc1405a248d39701963ce274a237 P e1881b44d0ad42d5b3c8cbe5e38aed5c: Tablet Copy: opened 5 blocks and 1 log segments I0507 06:33:27.285411 14660 ts_tablet_manager.cc:1377] T 6de7bc1405a248d39701963ce274a237 P 784329b2a1db4d4cb2e93c98aea55809: Deleting tablet data with delete state TABLET_DATA_COPYING I0507 06:33:27.289362 14660 ts_tablet_manager.cc:1390] T 6de7bc1405a248d39701963ce274a237 P 784329b2a1db4d4cb2e93c98aea55809: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.1 I0507 06:33:27.289695 14660 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet 6de7bc1405a248d39701963ce274a237. 1 dirs total, 0 dirs full, 0 dirs failed I0507 06:33:27.291990 14660 tablet_copy_client.cc:575] T 6de7bc1405a248d39701963ce274a237 P 784329b2a1db4d4cb2e93c98aea55809: tablet copy: Starting download of 5 data blocks... I0507 06:33:27.301470 14660 tablet_copy_client.cc:538] T 6de7bc1405a248d39701963ce274a237 P 784329b2a1db4d4cb2e93c98aea55809: tablet copy: Starting download of 1 WAL segments... I0507 06:33:27.304139 14660 tablet_copy_client.cc:414] T 6de7bc1405a248d39701963ce274a237 P 784329b2a1db4d4cb2e93c98aea55809: tablet copy: Tablet Copy complete. Replacing tablet superblock. I0507 06:33:27.307283 14660 tablet_bootstrap.cc:439] T 6de7bc1405a248d39701963ce274a237 P 784329b2a1db4d4cb2e93c98aea55809: Bootstrap starting. I0507 06:33:27.320140 14660 tablet_bootstrap.cc:439] T 6de7bc1405a248d39701963ce274a237 P 784329b2a1db4d4cb2e93c98aea55809: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=1} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=3. Pending: 0 replicates I0507 06:33:27.320551 14660 tablet_bootstrap.cc:439] T 6de7bc1405a248d39701963ce274a237 P 784329b2a1db4d4cb2e93c98aea55809: Bootstrap complete. I0507 06:33:27.320785 14660 ts_tablet_manager.cc:1053] T 6de7bc1405a248d39701963ce274a237 P 784329b2a1db4d4cb2e93c98aea55809: Time spent bootstrapping tablet: real 0.014s user 0.007s sys 0.009s I0507 06:33:27.321566 14660 raft_consensus.cc:340] T 6de7bc1405a248d39701963ce274a237 P 784329b2a1db4d4cb2e93c98aea55809 [term 1 LEARNER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: 39 OBSOLETE_local: false peers { permanent_uuid: "e1881b44d0ad42d5b3c8cbe5e38aed5c" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 41437 } } peers { permanent_uuid: "539eecf299464551a0697eeab25b54c2" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 45599 } } peers { permanent_uuid: "784329b2a1db4d4cb2e93c98aea55809" member_type: NON_VOTER last_known_addr { host: "127.0.57.193" port: 46571 } attrs { promote: true } } I0507 06:33:27.321774 14660 raft_consensus.cc:705] T 6de7bc1405a248d39701963ce274a237 P 784329b2a1db4d4cb2e93c98aea55809 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 784329b2a1db4d4cb2e93c98aea55809, State: Initialized, Role: LEARNER I0507 06:33:27.321936 14660 consensus_queue.cc:229] T 6de7bc1405a248d39701963ce274a237 P 784329b2a1db4d4cb2e93c98aea55809 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 39, Last appended: 1.39, Last appended by leader: 39, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 39 OBSOLETE_local: false peers { permanent_uuid: "e1881b44d0ad42d5b3c8cbe5e38aed5c" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 41437 } } peers { permanent_uuid: "539eecf299464551a0697eeab25b54c2" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 45599 } } peers { permanent_uuid: "784329b2a1db4d4cb2e93c98aea55809" member_type: NON_VOTER last_known_addr { host: "127.0.57.193" port: 46571 } attrs { promote: true } } I0507 06:33:27.322618 14567 heartbeater.cc:467] Master 127.0.57.254:44949 was elected leader, sending a full tablet report... I0507 06:33:27.322628 14660 ts_tablet_manager.cc:1081] T 6de7bc1405a248d39701963ce274a237 P 784329b2a1db4d4cb2e93c98aea55809: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s I0507 06:33:27.323545 14346 tablet_copy_service.cc:325] P e1881b44d0ad42d5b3c8cbe5e38aed5c: Request end of tablet copy session 784329b2a1db4d4cb2e93c98aea55809-6de7bc1405a248d39701963ce274a237 received from {username='slave'} at 127.0.57.193:50543 I0507 06:33:27.323681 14346 tablet_copy_service.cc:413] P e1881b44d0ad42d5b3c8cbe5e38aed5c: ending tablet copy session 784329b2a1db4d4cb2e93c98aea55809-6de7bc1405a248d39701963ce274a237 on tablet 6de7bc1405a248d39701963ce274a237 with peer 784329b2a1db4d4cb2e93c98aea55809 I0507 06:33:27.337730 14568 maintenance_manager.cc:306] P 784329b2a1db4d4cb2e93c98aea55809: Scheduling UndoDeltaBlockGCOp(6de7bc1405a248d39701963ce274a237): 536 bytes on disk I0507 06:33:27.339341 14459 maintenance_manager.cc:501] P 784329b2a1db4d4cb2e93c98aea55809: UndoDeltaBlockGCOp(6de7bc1405a248d39701963ce274a237) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":108,"lbm_reads_lt_1ms":4} I0507 06:33:27.741482 14524 raft_consensus.cc:1126] T 6de7bc1405a248d39701963ce274a237 P 784329b2a1db4d4cb2e93c98aea55809 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.38->[1.39-1.39] Dedup: 1.39->[] I0507 06:33:28.137923 14666 raft_consensus.cc:974] T 6de7bc1405a248d39701963ce274a237 P e1881b44d0ad42d5b3c8cbe5e38aed5c: attempting to promote NON_VOTER 784329b2a1db4d4cb2e93c98aea55809 to VOTER I0507 06:33:28.138592 14666 consensus_queue.cc:206] T 6de7bc1405a248d39701963ce274a237 P e1881b44d0ad42d5b3c8cbe5e38aed5c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 39, Committed index: 39, Last appended: 1.39, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 40 OBSOLETE_local: false peers { permanent_uuid: "e1881b44d0ad42d5b3c8cbe5e38aed5c" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 41437 } } peers { permanent_uuid: "539eecf299464551a0697eeab25b54c2" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 45599 } } peers { permanent_uuid: "784329b2a1db4d4cb2e93c98aea55809" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 46571 } attrs { promote: false } } I0507 06:33:28.140823 14204 raft_consensus.cc:1181] T 6de7bc1405a248d39701963ce274a237 P 539eecf299464551a0697eeab25b54c2 [term 1 FOLLOWER]: Refusing update from remote peer e1881b44d0ad42d5b3c8cbe5e38aed5c: Log matching property violated. Preceding OpId in replica: term: 1 index: 39. Preceding OpId from leader: term: 1 index: 40. (index mismatch) I0507 06:33:28.141106 14524 raft_consensus.cc:1181] T 6de7bc1405a248d39701963ce274a237 P 784329b2a1db4d4cb2e93c98aea55809 [term 1 LEARNER]: Refusing update from remote peer e1881b44d0ad42d5b3c8cbe5e38aed5c: Log matching property violated. Preceding OpId in replica: term: 1 index: 39. Preceding OpId from leader: term: 1 index: 40. (index mismatch) I0507 06:33:28.141381 14665 consensus_queue.cc:984] T 6de7bc1405a248d39701963ce274a237 P e1881b44d0ad42d5b3c8cbe5e38aed5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "539eecf299464551a0697eeab25b54c2" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 45599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 40, Last known committed idx: 39, Time since last communication: 0.000s I0507 06:33:28.141650 14654 consensus_queue.cc:984] T 6de7bc1405a248d39701963ce274a237 P e1881b44d0ad42d5b3c8cbe5e38aed5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "784329b2a1db4d4cb2e93c98aea55809" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 46571 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 40, Last known committed idx: 39, Time since last communication: 0.000s I0507 06:33:28.145112 14654 raft_consensus.cc:2788] T 6de7bc1405a248d39701963ce274a237 P e1881b44d0ad42d5b3c8cbe5e38aed5c [term 1 LEADER]: Committing config change with OpId 1.40: config changed from index 39 to 40, 784329b2a1db4d4cb2e93c98aea55809 (127.0.57.193) changed from NON_VOTER to VOTER. New config: { opid_index: 40 OBSOLETE_local: false peers { permanent_uuid: "e1881b44d0ad42d5b3c8cbe5e38aed5c" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 41437 } } peers { permanent_uuid: "539eecf299464551a0697eeab25b54c2" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 45599 } } peers { permanent_uuid: "784329b2a1db4d4cb2e93c98aea55809" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 46571 } attrs { promote: false } } } I0507 06:33:28.145817 14524 raft_consensus.cc:2788] T 6de7bc1405a248d39701963ce274a237 P 784329b2a1db4d4cb2e93c98aea55809 [term 1 FOLLOWER]: Committing config change with OpId 1.40: config changed from index 39 to 40, 784329b2a1db4d4cb2e93c98aea55809 (127.0.57.193) changed from NON_VOTER to VOTER. New config: { opid_index: 40 OBSOLETE_local: false peers { permanent_uuid: "e1881b44d0ad42d5b3c8cbe5e38aed5c" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 41437 } } peers { permanent_uuid: "539eecf299464551a0697eeab25b54c2" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 45599 } } peers { permanent_uuid: "784329b2a1db4d4cb2e93c98aea55809" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 46571 } attrs { promote: false } } } I0507 06:33:28.149749 14204 raft_consensus.cc:2788] T 6de7bc1405a248d39701963ce274a237 P 539eecf299464551a0697eeab25b54c2 [term 1 FOLLOWER]: Committing config change with OpId 1.40: config changed from index 39 to 40, 784329b2a1db4d4cb2e93c98aea55809 (127.0.57.193) changed from NON_VOTER to VOTER. New config: { opid_index: 40 OBSOLETE_local: false peers { permanent_uuid: "e1881b44d0ad42d5b3c8cbe5e38aed5c" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 41437 } } peers { permanent_uuid: "539eecf299464551a0697eeab25b54c2" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 45599 } } peers { permanent_uuid: "784329b2a1db4d4cb2e93c98aea55809" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 46571 } attrs { promote: false } } } I0507 06:33:28.151042 14597 catalog_manager.cc:3988] T 6de7bc1405a248d39701963ce274a237 P 784329b2a1db4d4cb2e93c98aea55809 reported cstate change: config changed from index 39 to 40, 784329b2a1db4d4cb2e93c98aea55809 (127.0.57.193) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e1881b44d0ad42d5b3c8cbe5e38aed5c" committed_config { opid_index: 40 OBSOLETE_local: false peers { permanent_uuid: "e1881b44d0ad42d5b3c8cbe5e38aed5c" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 41437 } } peers { permanent_uuid: "539eecf299464551a0697eeab25b54c2" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 45599 } } peers { permanent_uuid: "784329b2a1db4d4cb2e93c98aea55809" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 46571 } attrs { promote: false } } } Master Summary UUID | Address | Status ----------------------------------+--------------------+--------- 16571e2dbc9d46159c1792ce321cd0d7 | 127.0.57.254:44949 | HEALTHY Flag | Value | Tags | Master ------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+------------------------- ipki_ca_key_size | 1024 | experimental | all 1 server(s) checked ipki_server_key_size | 1024 | experimental | all 1 server(s) checked master_add_server_when_underreplicated | true | hidden | all 1 server(s) checked never_fsync | true | unsafe,advanced | all 1 server(s) checked raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked rpc_reuseport | true | experimental | all 1 server(s) checked rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked server_dump_info_format | pb | hidden | all 1 server(s) checked server_dump_info_path | /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_2.1557210500371863-231/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked Tablet Server Summary UUID | Address | Status | Location ----------------------------------+--------------------+---------+---------- 539eecf299464551a0697eeab25b54c2 | 127.0.57.194:45599 | HEALTHY | 784329b2a1db4d4cb2e93c98aea55809 | 127.0.57.193:46571 | HEALTHY | e1881b44d0ad42d5b3c8cbe5e38aed5c | 127.0.57.195:41437 | HEALTHY | Tablet Server Location Summary Location | Count ----------+--------- | 3 Flag | Value | Tags | Tablet Server ------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------------------------------+------------------------- ipki_server_key_size | 1024 | experimental | all 3 server(s) checked local_ip_for_outbound_sockets | 127.0.57.193 | experimental | 127.0.57.193:46571 local_ip_for_outbound_sockets | 127.0.57.194 | experimental | 127.0.57.194:45599 local_ip_for_outbound_sockets | 127.0.57.195 | experimental | 127.0.57.195:41437 log_compression_codec | none | experimental | all 3 server(s) checked log_max_segments_to_retain | 3 | experimental,advanced,runtime | all 3 server(s) checked log_target_replay_size_mb | 1 | experimental | all 3 server(s) checked maintenance_manager_polling_interval_ms | 100 | hidden | all 3 server(s) checked max_cell_size_bytes | 1000000 | unsafe | all 3 server(s) checked never_fsync | true | unsafe,advanced | all 3 server(s) checked raft_prepare_replacement_before_eviction | true | experimental,advanced | all 3 server(s) checked rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked server_dump_info_format | pb | hidden | all 3 server(s) checked server_dump_info_path | /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_2.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.0.57.193:46571 server_dump_info_path | /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_2.1557210500371863-231/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.0.57.194:45599 server_dump_info_path | /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_2.1557210500371863-231/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.0.57.195:41437 Version Summary Version | Servers ------------------------+------------------------- 1.10.0-cdh6.x-SNAPSHOT | all 4 server(s) checked Tablet Summary Summary by table Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable -----------+----+---------+---------------+---------+------------+------------------+------------- TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0 Tablet Replica Count Summary Statistic | Replica Count ----------------+--------------- Minimum | 1 First Quartile | 1 Median | 1 Third Quartile | 1 Maximum | 1 Total Count Summary | Total Count ----------------+------------- Masters | 1 Tablet Servers | 3 Tables | 1 Tablets | 1 Replicas | 3 ================== Warnings: ================== Some masters have unsafe, experimental, or hidden flags set Some tablet servers have unsafe, experimental, or hidden flags set OK I0507 06:33:28.235528 231 log_verifier.cc:121] Checking tablet 6de7bc1405a248d39701963ce274a237 I0507 06:33:28.267012 231 log_verifier.cc:172] Verified matching terms for 7 ops in tablet 6de7bc1405a248d39701963ce274a237 I0507 06:33:28.267953 231 external_mini_cluster.cc:1172] Attempting to check leaks for /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver pid 14445 I0507 06:33:28.273310 14564 generic_service.cc:171] Checking for leaks (request via RPC) I0507 06:33:28.331634 231 external_mini_cluster.cc:1131] Killing /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver with pid 14445 I0507 06:33:28.341526 231 external_mini_cluster.cc:1172] Attempting to check leaks for /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver pid 14128 I0507 06:33:28.342586 14244 generic_service.cc:171] Checking for leaks (request via RPC) I0507 06:33:28.426195 231 external_mini_cluster.cc:1131] Killing /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver with pid 14128 I0507 06:33:28.442226 231 external_mini_cluster.cc:1172] Attempting to check leaks for /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver pid 14250 I0507 06:33:28.443200 14366 generic_service.cc:171] Checking for leaks (request via RPC) I0507 06:33:28.542804 231 external_mini_cluster.cc:1131] Killing /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver with pid 14250 I0507 06:33:28.561435 231 external_mini_cluster.cc:1172] Attempting to check leaks for /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-master pid 14573 I0507 06:33:28.562659 14629 generic_service.cc:171] Checking for leaks (request via RPC) I0507 06:33:28.638994 231 external_mini_cluster.cc:1131] Killing /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-master with pid 14573 [ OK ] ReplicaBehindWalGcThresholdITest.ReplicaReplacement/2 (8338 ms) [ RUN ] ReplicaBehindWalGcThresholdITest.ReplicaReplacement/3 I0507 06:33:28.662509 231 test_util.cc:212] Using random seed: 1696020960 I0507 06:33:28.663929 231 ts_itest-base.cc:112] Starting cluster with: I0507 06:33:28.663975 231 ts_itest-base.cc:113] -------------- I0507 06:33:28.663985 231 ts_itest-base.cc:114] 3 tablet servers I0507 06:33:28.663995 231 ts_itest-base.cc:115] 3 replicas per TS I0507 06:33:28.664002 231 ts_itest-base.cc:116] -------------- I0507 06:33:28.665093 231 external_mini_cluster.cc:933] Running /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-master /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-master --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_3.1557210500371863-231/raft_consensus-itest-cluster/master-0/wal --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_3.1557210500371863-231/raft_consensus-itest-cluster/master-0/data --block_manager=log --webserver_interface=localhost --ipki_ca_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.0.57.254:43845 --webserver_interface=127.0.57.254 --webserver_port=0 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_3.1557210500371863-231/raft_consensus-itest-cluster/master-0/logs --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_3.1557210500371863-231/raft_consensus-itest-cluster/master-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=true --master_add_server_when_underreplicated=false --rpc_reuseport=true with env {} WARNING: Logging before InitGoogleLogging() is written to STDERR W0507 06:33:28.874255 14711 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0507 06:33:28.874480 14711 flags.cc:404] Enabled unsafe flag: --never_fsync=true W0507 06:33:28.876904 14711 flags.cc:404] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true W0507 06:33:28.876971 14711 flags.cc:404] Enabled experimental flag: --ipki_ca_key_size=1024 W0507 06:33:28.876999 14711 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W0507 06:33:28.877032 14711 flags.cc:404] Enabled experimental flag: --tsk_num_rsa_bits=512 W0507 06:33:28.877069 14711 flags.cc:404] Enabled experimental flag: --rpc_reuseport=true I0507 06:33:28.881409 14711 master_main.cc:73] Master server non-default flags: --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_3.1557210500371863-231/raft_consensus-itest-cluster/master-0/data --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_3.1557210500371863-231/raft_consensus-itest-cluster/master-0/wal --master_add_server_when_underreplicated=false --ipki_ca_key_size=1024 --ipki_server_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.0.57.254:43845 --rpc_reuseport=true --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_3.1557210500371863-231/raft_consensus-itest-cluster/master-0/data/info.pb --webserver_interface=127.0.57.254 --webserver_port=0 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_3.1557210500371863-231/raft_consensus-itest-cluster/master-0/logs --logbuflevel=-1 --logtostderr=true Master server version: kudu 1.10.0-cdh6.x-SNAPSHOT revision 7d0ee26697926e87af9ef395008799577f6ab43f build type FASTDEBUG built by jenkins at 06 May 2019 22:44:21 PST on quasar-knmpfx-1.vpc.cloudera.com build id 322017 ASAN enabled I0507 06:33:28.887886 14711 system_ntp.cc:202] NTP initialized. Skew: 500ppm Current error: 178516us I0507 06:33:28.889158 14711 fs_manager.cc:263] Metadata directory not provided I0507 06:33:28.889228 14711 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0507 06:33:28.889503 14711 server_base.cc:433] Could not load existing FS layout: Not found: could not find a healthy instance file I0507 06:33:28.889566 14711 server_base.cc:434] Attempting to create new FS layout instead I0507 06:33:28.892025 14711 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_3.1557210500371863-231/raft_consensus-itest-cluster/master-0/data/instance: uuid: "4b3cf1ac833f41f8ba8fb8623982ab03" format_stamp: "Formatted at 2019-05-07 06:33:28 on dist-test-slave-dist-test-slave-rqp6" I0507 06:33:28.892552 14711 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_3.1557210500371863-231/raft_consensus-itest-cluster/master-0/wal/instance: uuid: "4b3cf1ac833f41f8ba8fb8623982ab03" format_stamp: "Formatted at 2019-05-07 06:33:28 on dist-test-slave-dist-test-slave-rqp6" I0507 06:33:28.899737 14711 fs_manager.cc:503] Time spent creating directory manager: real 0.007s user 0.004s sys 0.000s I0507 06:33:28.901165 14711 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0507 06:33:28.901927 14711 file_cache.cc:466] Constructed file cache lbm with capacity 419430 I0507 06:33:28.905550 14711 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s I0507 06:33:28.905658 14711 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_3.1557210500371863-231/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_3.1557210500371863-231/raft_consensus-itest-cluster/master-0/wal uuid: "4b3cf1ac833f41f8ba8fb8623982ab03" format_stamp: "Formatted at 2019-05-07 06:33:28 on dist-test-slave-dist-test-slave-rqp6" I0507 06:33:28.905838 14711 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_3.1557210500371863-231/raft_consensus-itest-cluster/master-0/wal metadata directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_3.1557210500371863-231/raft_consensus-itest-cluster/master-0/wal 1 data directories: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_3.1557210500371863-231/raft_consensus-itest-cluster/master-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0507 06:33:28.931517 14711 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0507 06:33:28.946027 14711 rpc_server.cc:205] RPC server started. Bound to: 127.0.57.254:43845 I0507 06:33:28.947368 14711 webserver.cc:308] Webserver started at http://127.0.57.254:35109/ using document root and password file I0507 06:33:28.948206 14711 server_base.cc:608] Dumped server information to /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_3.1557210500371863-231/raft_consensus-itest-cluster/master-0/data/info.pb I0507 06:33:28.952042 14770 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0507 06:33:28.952721 231 external_mini_cluster.cc:995] Started /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-master as pid 14711 I0507 06:33:28.953438 231 external_mini_cluster.cc:933] Running /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_3.1557210500371863-231/raft_consensus-itest-cluster/ts-0/wal --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_3.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data --block_manager=log --rpc_bind_addresses=127.0.57.193:0 --local_ip_for_outbound_sockets=127.0.57.193 --webserver_interface=127.0.57.193 --webserver_port=0 --tserver_master_addrs=127.0.57.254:43845 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_3.1557210500371863-231/raft_consensus-itest-cluster/ts-0/logs --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_3.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=true --follower_unavailable_considered_failed_sec=5 --log_compression_codec=none --log_cache_size_limit_mb=1 --log_segment_size_mb=1 --log_async_preallocate_segments=false --log_min_segments_to_retain=1 --log_max_segments_to_retain=3 --maintenance_manager_polling_interval_ms=100 --log_target_replay_size_mb=1 --max_cell_size_bytes=1000000 with env {} I0507 06:33:28.963326 14770 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 4b3cf1ac833f41f8ba8fb8623982ab03: Bootstrap starting. I0507 06:33:28.966203 14770 tablet_bootstrap.cc:590] T 00000000000000000000000000000000 P 4b3cf1ac833f41f8ba8fb8623982ab03: No blocks or log segments found. Creating new log. I0507 06:33:28.967139 14770 log.cc:525] T 00000000000000000000000000000000 P 4b3cf1ac833f41f8ba8fb8623982ab03: Log is configured to *not* fsync() on all Append() calls I0507 06:33:28.968839 14770 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 4b3cf1ac833f41f8ba8fb8623982ab03: No bootstrap required, opened a new log I0507 06:33:28.975294 14770 raft_consensus.cc:340] T 00000000000000000000000000000000 P 4b3cf1ac833f41f8ba8fb8623982ab03 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b3cf1ac833f41f8ba8fb8623982ab03" member_type: VOTER } I0507 06:33:28.975525 14770 raft_consensus.cc:366] T 00000000000000000000000000000000 P 4b3cf1ac833f41f8ba8fb8623982ab03 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0507 06:33:28.975577 14770 raft_consensus.cc:705] T 00000000000000000000000000000000 P 4b3cf1ac833f41f8ba8fb8623982ab03 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b3cf1ac833f41f8ba8fb8623982ab03, State: Initialized, Role: FOLLOWER I0507 06:33:28.976073 14770 consensus_queue.cc:229] T 00000000000000000000000000000000 P 4b3cf1ac833f41f8ba8fb8623982ab03 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b3cf1ac833f41f8ba8fb8623982ab03" member_type: VOTER } I0507 06:33:28.976339 14770 raft_consensus.cc:380] T 00000000000000000000000000000000 P 4b3cf1ac833f41f8ba8fb8623982ab03 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0507 06:33:28.976423 14770 raft_consensus.cc:472] T 00000000000000000000000000000000 P 4b3cf1ac833f41f8ba8fb8623982ab03 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0507 06:33:28.976513 14770 raft_consensus.cc:2886] T 00000000000000000000000000000000 P 4b3cf1ac833f41f8ba8fb8623982ab03 [term 0 FOLLOWER]: Advancing to term 1 I0507 06:33:28.978102 14770 raft_consensus.cc:494] T 00000000000000000000000000000000 P 4b3cf1ac833f41f8ba8fb8623982ab03 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b3cf1ac833f41f8ba8fb8623982ab03" member_type: VOTER } I0507 06:33:28.978520 14770 leader_election.cc:310] T 00000000000000000000000000000000 P 4b3cf1ac833f41f8ba8fb8623982ab03 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4b3cf1ac833f41f8ba8fb8623982ab03; no voters: I0507 06:33:28.979279 14773 raft_consensus.cc:2640] T 00000000000000000000000000000000 P 4b3cf1ac833f41f8ba8fb8623982ab03 [term 1 FOLLOWER]: Leader election won for term 1 I0507 06:33:28.979589 14770 leader_election.cc:296] T 00000000000000000000000000000000 P 4b3cf1ac833f41f8ba8fb8623982ab03 [CANDIDATE]: Term 1 election: Requested vote from peers I0507 06:33:28.980348 14773 raft_consensus.cc:667] T 00000000000000000000000000000000 P 4b3cf1ac833f41f8ba8fb8623982ab03 [term 1 LEADER]: Becoming Leader. State: Replica: 4b3cf1ac833f41f8ba8fb8623982ab03, State: Running, Role: LEADER I0507 06:33:28.980751 14773 consensus_queue.cc:206] T 00000000000000000000000000000000 P 4b3cf1ac833f41f8ba8fb8623982ab03 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b3cf1ac833f41f8ba8fb8623982ab03" member_type: VOTER } I0507 06:33:28.982549 14770 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4b3cf1ac833f41f8ba8fb8623982ab03 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 1 leader_uuid: "4b3cf1ac833f41f8ba8fb8623982ab03" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b3cf1ac833f41f8ba8fb8623982ab03" member_type: VOTER } } I0507 06:33:28.983206 14770 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4b3cf1ac833f41f8ba8fb8623982ab03 [sys.catalog]: This master's current role is: LEADER I0507 06:33:28.983422 14775 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4b3cf1ac833f41f8ba8fb8623982ab03 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4b3cf1ac833f41f8ba8fb8623982ab03. Latest consensus state: current_term: 1 leader_uuid: "4b3cf1ac833f41f8ba8fb8623982ab03" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b3cf1ac833f41f8ba8fb8623982ab03" member_type: VOTER } } I0507 06:33:28.984015 14775 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4b3cf1ac833f41f8ba8fb8623982ab03 [sys.catalog]: This master's current role is: LEADER I0507 06:33:28.983655 14774 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4b3cf1ac833f41f8ba8fb8623982ab03 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4b3cf1ac833f41f8ba8fb8623982ab03" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b3cf1ac833f41f8ba8fb8623982ab03" member_type: VOTER } } I0507 06:33:28.984685 14774 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4b3cf1ac833f41f8ba8fb8623982ab03 [sys.catalog]: This master's current role is: LEADER I0507 06:33:28.986114 14770 sys_catalog.cc:424] T 00000000000000000000000000000000 P 4b3cf1ac833f41f8ba8fb8623982ab03 [sys.catalog]: configured and running, proceeding with master startup. I0507 06:33:28.986323 14777 catalog_manager.cc:1055] Loading table and tablet metadata into memory... I0507 06:33:28.991009 14777 catalog_manager.cc:1066] Initializing Kudu internal certificate authority... I0507 06:33:29.045199 14777 catalog_manager.cc:950] Generated new certificate authority record I0507 06:33:29.046797 14777 catalog_manager.cc:1075] Loading token signing keys... I0507 06:33:29.060561 14777 catalog_manager.cc:4292] T 00000000000000000000000000000000 P 4b3cf1ac833f41f8ba8fb8623982ab03: Generated new TSK 0 WARNING: Logging before InitGoogleLogging() is written to STDERR W0507 06:33:29.138561 14771 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0507 06:33:29.138811 14771 flags.cc:404] Enabled unsafe flag: --max_cell_size_bytes=1000000 W0507 06:33:29.138859 14771 flags.cc:404] Enabled unsafe flag: --never_fsync=true W0507 06:33:29.141067 14771 flags.cc:404] Enabled experimental flag: --log_compression_codec=none W0507 06:33:29.141127 14771 flags.cc:404] Enabled experimental flag: --log_max_segments_to_retain=3 W0507 06:33:29.141165 14771 flags.cc:404] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true W0507 06:33:29.141227 14771 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W0507 06:33:29.141311 14771 flags.cc:404] Enabled experimental flag: --log_target_replay_size_mb=1 W0507 06:33:29.141367 14771 flags.cc:404] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.57.193 I0507 06:33:29.156714 14771 tablet_server_main.cc:78] Tablet server non-default flags: --follower_unavailable_considered_failed_sec=5 --log_compression_codec=none --log_max_segments_to_retain=3 --log_cache_size_limit_mb=1 --log_async_preallocate_segments=false --log_segment_size_mb=1 --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_3.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_3.1557210500371863-231/raft_consensus-itest-cluster/ts-0/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.0.57.193:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_3.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data/info.pb --webserver_interface=127.0.57.193 --webserver_port=0 --max_cell_size_bytes=1000000 --tserver_master_addrs=127.0.57.254:43845 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --log_target_replay_size_mb=1 --maintenance_manager_polling_interval_ms=100 --enable_minidumps=false --local_ip_for_outbound_sockets=127.0.57.193 --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_3.1557210500371863-231/raft_consensus-itest-cluster/ts-0/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.10.0-cdh6.x-SNAPSHOT revision 7d0ee26697926e87af9ef395008799577f6ab43f build type FASTDEBUG built by jenkins at 06 May 2019 22:44:21 PST on quasar-knmpfx-1.vpc.cloudera.com build id 322017 ASAN enabled I0507 06:33:29.165853 14771 system_ntp.cc:202] NTP initialized. Skew: 500ppm Current error: 179016us I0507 06:33:29.166671 14771 fs_manager.cc:263] Metadata directory not provided I0507 06:33:29.166756 14771 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0507 06:33:29.167070 14771 server_base.cc:433] Could not load existing FS layout: Not found: could not find a healthy instance file I0507 06:33:29.167143 14771 server_base.cc:434] Attempting to create new FS layout instead I0507 06:33:29.169822 14771 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_3.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data/instance: uuid: "69ac1841085d4196ac900d65a0238f79" format_stamp: "Formatted at 2019-05-07 06:33:29 on dist-test-slave-dist-test-slave-rqp6" I0507 06:33:29.170413 14771 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_3.1557210500371863-231/raft_consensus-itest-cluster/ts-0/wal/instance: uuid: "69ac1841085d4196ac900d65a0238f79" format_stamp: "Formatted at 2019-05-07 06:33:29 on dist-test-slave-dist-test-slave-rqp6" I0507 06:33:29.179980 14771 fs_manager.cc:503] Time spent creating directory manager: real 0.009s user 0.005s sys 0.005s I0507 06:33:29.181295 14771 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0507 06:33:29.182005 14771 file_cache.cc:466] Constructed file cache lbm with capacity 419430 I0507 06:33:29.185750 14771 fs_manager.cc:419] Time spent opening block manager: real 0.003s user 0.000s sys 0.000s I0507 06:33:29.185871 14771 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_3.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_3.1557210500371863-231/raft_consensus-itest-cluster/ts-0/wal uuid: "69ac1841085d4196ac900d65a0238f79" format_stamp: "Formatted at 2019-05-07 06:33:29 on dist-test-slave-dist-test-slave-rqp6" I0507 06:33:29.186081 14771 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_3.1557210500371863-231/raft_consensus-itest-cluster/ts-0/wal metadata directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_3.1557210500371863-231/raft_consensus-itest-cluster/ts-0/wal 1 data directories: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_3.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0507 06:33:29.229841 14771 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0507 06:33:29.231858 14771 ts_tablet_manager.cc:356] Loaded tablet metadata (0 total tablets, 0 live tablets) I0507 06:33:29.231936 14771 ts_tablet_manager.cc:376] Registered 0 tablets I0507 06:33:29.291839 14771 rpc_server.cc:205] RPC server started. Bound to: 127.0.57.193:43169 I0507 06:33:29.293292 14771 webserver.cc:308] Webserver started at http://127.0.57.193:39453/ using document root and password file I0507 06:33:29.294206 14771 server_base.cc:608] Dumped server information to /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_3.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data/info.pb I0507 06:33:29.300416 231 external_mini_cluster.cc:995] Started /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver as pid 14771 I0507 06:33:29.301059 231 external_mini_cluster.cc:933] Running /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_3.1557210500371863-231/raft_consensus-itest-cluster/ts-1/wal --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_3.1557210500371863-231/raft_consensus-itest-cluster/ts-1/data --block_manager=log --rpc_bind_addresses=127.0.57.194:0 --local_ip_for_outbound_sockets=127.0.57.194 --webserver_interface=127.0.57.194 --webserver_port=0 --tserver_master_addrs=127.0.57.254:43845 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_3.1557210500371863-231/raft_consensus-itest-cluster/ts-1/logs --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_3.1557210500371863-231/raft_consensus-itest-cluster/ts-1/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=true --follower_unavailable_considered_failed_sec=5 --log_compression_codec=none --log_cache_size_limit_mb=1 --log_segment_size_mb=1 --log_async_preallocate_segments=false --log_min_segments_to_retain=1 --log_max_segments_to_retain=3 --maintenance_manager_polling_interval_ms=100 --log_target_replay_size_mb=1 --max_cell_size_bytes=1000000 with env {} I0507 06:33:29.312733 14900 heartbeater.cc:345] Connected to a master server at 127.0.57.254:43845 I0507 06:33:29.312978 14900 heartbeater.cc:426] Registering TS with master... I0507 06:33:29.313555 14900 heartbeater.cc:475] Master 127.0.57.254:43845 requested a full tablet report, sending... I0507 06:33:29.315415 14737 ts_manager.cc:159] Registered new tserver with Master: 69ac1841085d4196ac900d65a0238f79 (127.0.57.193:43169) I0507 06:33:29.317034 14737 master_service.cc:273] Signed X509 certificate for tserver {username='slave'} at 127.0.57.193:49883 WARNING: Logging before InitGoogleLogging() is written to STDERR W0507 06:33:29.480010 14904 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0507 06:33:29.480234 14904 flags.cc:404] Enabled unsafe flag: --max_cell_size_bytes=1000000 W0507 06:33:29.480278 14904 flags.cc:404] Enabled unsafe flag: --never_fsync=true W0507 06:33:29.482537 14904 flags.cc:404] Enabled experimental flag: --log_compression_codec=none W0507 06:33:29.482595 14904 flags.cc:404] Enabled experimental flag: --log_max_segments_to_retain=3 W0507 06:33:29.482633 14904 flags.cc:404] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true W0507 06:33:29.482676 14904 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W0507 06:33:29.482779 14904 flags.cc:404] Enabled experimental flag: --log_target_replay_size_mb=1 W0507 06:33:29.482849 14904 flags.cc:404] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.57.194 I0507 06:33:29.487260 14904 tablet_server_main.cc:78] Tablet server non-default flags: --follower_unavailable_considered_failed_sec=5 --log_compression_codec=none --log_max_segments_to_retain=3 --log_cache_size_limit_mb=1 --log_async_preallocate_segments=false --log_segment_size_mb=1 --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_3.1557210500371863-231/raft_consensus-itest-cluster/ts-1/data --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_3.1557210500371863-231/raft_consensus-itest-cluster/ts-1/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.0.57.194:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_3.1557210500371863-231/raft_consensus-itest-cluster/ts-1/data/info.pb --webserver_interface=127.0.57.194 --webserver_port=0 --max_cell_size_bytes=1000000 --tserver_master_addrs=127.0.57.254:43845 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --log_target_replay_size_mb=1 --maintenance_manager_polling_interval_ms=100 --enable_minidumps=false --local_ip_for_outbound_sockets=127.0.57.194 --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_3.1557210500371863-231/raft_consensus-itest-cluster/ts-1/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.10.0-cdh6.x-SNAPSHOT revision 7d0ee26697926e87af9ef395008799577f6ab43f build type FASTDEBUG built by jenkins at 06 May 2019 22:44:21 PST on quasar-knmpfx-1.vpc.cloudera.com build id 322017 ASAN enabled I0507 06:33:29.495455 14904 system_ntp.cc:202] NTP initialized. Skew: 500ppm Current error: 179016us I0507 06:33:29.496189 14904 fs_manager.cc:263] Metadata directory not provided I0507 06:33:29.496242 14904 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0507 06:33:29.496574 14904 server_base.cc:433] Could not load existing FS layout: Not found: could not find a healthy instance file I0507 06:33:29.496645 14904 server_base.cc:434] Attempting to create new FS layout instead I0507 06:33:29.499161 14904 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_3.1557210500371863-231/raft_consensus-itest-cluster/ts-1/data/instance: uuid: "9d7b3cf9bef345e38290f5f6ec1b3a12" format_stamp: "Formatted at 2019-05-07 06:33:29 on dist-test-slave-dist-test-slave-rqp6" I0507 06:33:29.499821 14904 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_3.1557210500371863-231/raft_consensus-itest-cluster/ts-1/wal/instance: uuid: "9d7b3cf9bef345e38290f5f6ec1b3a12" format_stamp: "Formatted at 2019-05-07 06:33:29 on dist-test-slave-dist-test-slave-rqp6" I0507 06:33:29.506920 14904 fs_manager.cc:503] Time spent creating directory manager: real 0.007s user 0.007s sys 0.002s I0507 06:33:29.508419 14904 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0507 06:33:29.509200 14904 file_cache.cc:466] Constructed file cache lbm with capacity 419430 I0507 06:33:29.513195 14904 fs_manager.cc:419] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s I0507 06:33:29.513310 14904 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_3.1557210500371863-231/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_3.1557210500371863-231/raft_consensus-itest-cluster/ts-1/wal uuid: "9d7b3cf9bef345e38290f5f6ec1b3a12" format_stamp: "Formatted at 2019-05-07 06:33:29 on dist-test-slave-dist-test-slave-rqp6" I0507 06:33:29.513522 14904 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_3.1557210500371863-231/raft_consensus-itest-cluster/ts-1/wal metadata directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_3.1557210500371863-231/raft_consensus-itest-cluster/ts-1/wal 1 data directories: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_3.1557210500371863-231/raft_consensus-itest-cluster/ts-1/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0507 06:33:29.548840 14904 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0507 06:33:29.550700 14904 ts_tablet_manager.cc:356] Loaded tablet metadata (0 total tablets, 0 live tablets) I0507 06:33:29.550786 14904 ts_tablet_manager.cc:376] Registered 0 tablets I0507 06:33:29.576588 14904 rpc_server.cc:205] RPC server started. Bound to: 127.0.57.194:38001 I0507 06:33:29.577845 14904 webserver.cc:308] Webserver started at http://127.0.57.194:34491/ using document root and password file I0507 06:33:29.578521 14904 server_base.cc:608] Dumped server information to /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_3.1557210500371863-231/raft_consensus-itest-cluster/ts-1/data/info.pb I0507 06:33:29.588624 231 external_mini_cluster.cc:995] Started /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver as pid 14904 I0507 06:33:29.589341 231 external_mini_cluster.cc:933] Running /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_3.1557210500371863-231/raft_consensus-itest-cluster/ts-2/wal --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_3.1557210500371863-231/raft_consensus-itest-cluster/ts-2/data --block_manager=log --rpc_bind_addresses=127.0.57.195:0 --local_ip_for_outbound_sockets=127.0.57.195 --webserver_interface=127.0.57.195 --webserver_port=0 --tserver_master_addrs=127.0.57.254:43845 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_3.1557210500371863-231/raft_consensus-itest-cluster/ts-2/logs --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_3.1557210500371863-231/raft_consensus-itest-cluster/ts-2/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=true --follower_unavailable_considered_failed_sec=5 --log_compression_codec=none --log_cache_size_limit_mb=1 --log_segment_size_mb=1 --log_async_preallocate_segments=false --log_min_segments_to_retain=1 --log_max_segments_to_retain=3 --maintenance_manager_polling_interval_ms=100 --log_target_replay_size_mb=1 --max_cell_size_bytes=1000000 with env {} I0507 06:33:29.592703 15023 heartbeater.cc:345] Connected to a master server at 127.0.57.254:43845 I0507 06:33:29.592944 15023 heartbeater.cc:426] Registering TS with master... I0507 06:33:29.593482 15023 heartbeater.cc:475] Master 127.0.57.254:43845 requested a full tablet report, sending... I0507 06:33:29.594542 14737 ts_manager.cc:159] Registered new tserver with Master: 9d7b3cf9bef345e38290f5f6ec1b3a12 (127.0.57.194:38001) I0507 06:33:29.595427 14737 master_service.cc:273] Signed X509 certificate for tserver {username='slave'} at 127.0.57.194:51449 WARNING: Logging before InitGoogleLogging() is written to STDERR W0507 06:33:29.768887 15026 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0507 06:33:29.769148 15026 flags.cc:404] Enabled unsafe flag: --max_cell_size_bytes=1000000 W0507 06:33:29.769196 15026 flags.cc:404] Enabled unsafe flag: --never_fsync=true W0507 06:33:29.771327 15026 flags.cc:404] Enabled experimental flag: --log_compression_codec=none W0507 06:33:29.771394 15026 flags.cc:404] Enabled experimental flag: --log_max_segments_to_retain=3 W0507 06:33:29.771478 15026 flags.cc:404] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true W0507 06:33:29.771518 15026 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W0507 06:33:29.771612 15026 flags.cc:404] Enabled experimental flag: --log_target_replay_size_mb=1 W0507 06:33:29.771656 15026 flags.cc:404] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.57.195 I0507 06:33:29.775981 15026 tablet_server_main.cc:78] Tablet server non-default flags: --follower_unavailable_considered_failed_sec=5 --log_compression_codec=none --log_max_segments_to_retain=3 --log_cache_size_limit_mb=1 --log_async_preallocate_segments=false --log_segment_size_mb=1 --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_3.1557210500371863-231/raft_consensus-itest-cluster/ts-2/data --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_3.1557210500371863-231/raft_consensus-itest-cluster/ts-2/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.0.57.195:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_3.1557210500371863-231/raft_consensus-itest-cluster/ts-2/data/info.pb --webserver_interface=127.0.57.195 --webserver_port=0 --max_cell_size_bytes=1000000 --tserver_master_addrs=127.0.57.254:43845 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --log_target_replay_size_mb=1 --maintenance_manager_polling_interval_ms=100 --enable_minidumps=false --local_ip_for_outbound_sockets=127.0.57.195 --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_3.1557210500371863-231/raft_consensus-itest-cluster/ts-2/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.10.0-cdh6.x-SNAPSHOT revision 7d0ee26697926e87af9ef395008799577f6ab43f build type FASTDEBUG built by jenkins at 06 May 2019 22:44:21 PST on quasar-knmpfx-1.vpc.cloudera.com build id 322017 ASAN enabled I0507 06:33:29.784615 15026 system_ntp.cc:202] NTP initialized. Skew: 500ppm Current error: 179016us I0507 06:33:29.785368 15026 fs_manager.cc:263] Metadata directory not provided I0507 06:33:29.785436 15026 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0507 06:33:29.785773 15026 server_base.cc:433] Could not load existing FS layout: Not found: could not find a healthy instance file I0507 06:33:29.785845 15026 server_base.cc:434] Attempting to create new FS layout instead I0507 06:33:29.788445 15026 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_3.1557210500371863-231/raft_consensus-itest-cluster/ts-2/data/instance: uuid: "8c9db460feac4f3ea7d87157753cc05f" format_stamp: "Formatted at 2019-05-07 06:33:29 on dist-test-slave-dist-test-slave-rqp6" I0507 06:33:29.789100 15026 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_3.1557210500371863-231/raft_consensus-itest-cluster/ts-2/wal/instance: uuid: "8c9db460feac4f3ea7d87157753cc05f" format_stamp: "Formatted at 2019-05-07 06:33:29 on dist-test-slave-dist-test-slave-rqp6" I0507 06:33:29.796408 15026 fs_manager.cc:503] Time spent creating directory manager: real 0.007s user 0.006s sys 0.000s I0507 06:33:29.797639 15026 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0507 06:33:29.798305 15026 file_cache.cc:466] Constructed file cache lbm with capacity 419430 I0507 06:33:29.801723 15026 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s I0507 06:33:29.801817 15026 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_3.1557210500371863-231/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_3.1557210500371863-231/raft_consensus-itest-cluster/ts-2/wal uuid: "8c9db460feac4f3ea7d87157753cc05f" format_stamp: "Formatted at 2019-05-07 06:33:29 on dist-test-slave-dist-test-slave-rqp6" I0507 06:33:29.802006 15026 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_3.1557210500371863-231/raft_consensus-itest-cluster/ts-2/wal metadata directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_3.1557210500371863-231/raft_consensus-itest-cluster/ts-2/wal 1 data directories: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_3.1557210500371863-231/raft_consensus-itest-cluster/ts-2/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0507 06:33:29.827073 15026 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0507 06:33:29.828913 15026 ts_tablet_manager.cc:356] Loaded tablet metadata (0 total tablets, 0 live tablets) I0507 06:33:29.828999 15026 ts_tablet_manager.cc:376] Registered 0 tablets I0507 06:33:29.852819 15026 rpc_server.cc:205] RPC server started. Bound to: 127.0.57.195:38149 I0507 06:33:29.854007 15026 webserver.cc:308] Webserver started at http://127.0.57.195:36001/ using document root and password file I0507 06:33:29.854709 15026 server_base.cc:608] Dumped server information to /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_3.1557210500371863-231/raft_consensus-itest-cluster/ts-2/data/info.pb I0507 06:33:29.864691 231 external_mini_cluster.cc:995] Started /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver as pid 15026 I0507 06:33:29.874423 15145 heartbeater.cc:345] Connected to a master server at 127.0.57.254:43845 I0507 06:33:29.874635 15145 heartbeater.cc:426] Registering TS with master... I0507 06:33:29.875157 15145 heartbeater.cc:475] Master 127.0.57.254:43845 requested a full tablet report, sending... I0507 06:33:29.876245 14737 ts_manager.cc:159] Registered new tserver with Master: 8c9db460feac4f3ea7d87157753cc05f (127.0.57.195:38149) I0507 06:33:29.877185 14737 master_service.cc:273] Signed X509 certificate for tserver {username='slave'} at 127.0.57.195:51631 I0507 06:33:29.878168 231 external_mini_cluster.cc:623] 3 TS(s) registered with all masters I0507 06:33:29.888280 14737 catalog_manager.cc:1415] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45234: name: "TestTable" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 3 split_rows_range_bounds { } partition_schema { range_schema { columns { name: "key" } } } W0507 06:33:29.893008 14737 catalog_manager.cc:1574] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestTable in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers. I0507 06:33:29.921975 14960 tablet_service.cc:1016] Processing CreateTablet for tablet 0712c795dc4648ffa156a17cf52d071c (table=TestTable [id=7222e5f199b649a0876b51c5387cb8a9]), partition=RANGE (key) PARTITION UNBOUNDED I0507 06:33:29.921973 15082 tablet_service.cc:1016] Processing CreateTablet for tablet 0712c795dc4648ffa156a17cf52d071c (table=TestTable [id=7222e5f199b649a0876b51c5387cb8a9]), partition=RANGE (key) PARTITION UNBOUNDED I0507 06:33:29.921941 14837 tablet_service.cc:1016] Processing CreateTablet for tablet 0712c795dc4648ffa156a17cf52d071c (table=TestTable [id=7222e5f199b649a0876b51c5387cb8a9]), partition=RANGE (key) PARTITION UNBOUNDED I0507 06:33:29.922840 15082 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet 0712c795dc4648ffa156a17cf52d071c. 1 dirs total, 0 dirs full, 0 dirs failed I0507 06:33:29.922842 14960 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet 0712c795dc4648ffa156a17cf52d071c. 1 dirs total, 0 dirs full, 0 dirs failed I0507 06:33:29.922881 14837 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet 0712c795dc4648ffa156a17cf52d071c. 1 dirs total, 0 dirs full, 0 dirs failed I0507 06:33:29.932160 15164 tablet_bootstrap.cc:439] T 0712c795dc4648ffa156a17cf52d071c P 8c9db460feac4f3ea7d87157753cc05f: Bootstrap starting. I0507 06:33:29.933013 15165 tablet_bootstrap.cc:439] T 0712c795dc4648ffa156a17cf52d071c P 9d7b3cf9bef345e38290f5f6ec1b3a12: Bootstrap starting. I0507 06:33:29.936075 15164 tablet_bootstrap.cc:590] T 0712c795dc4648ffa156a17cf52d071c P 8c9db460feac4f3ea7d87157753cc05f: No blocks or log segments found. Creating new log. I0507 06:33:29.937788 15165 tablet_bootstrap.cc:590] T 0712c795dc4648ffa156a17cf52d071c P 9d7b3cf9bef345e38290f5f6ec1b3a12: No blocks or log segments found. Creating new log. I0507 06:33:29.938664 15165 log.cc:525] T 0712c795dc4648ffa156a17cf52d071c P 9d7b3cf9bef345e38290f5f6ec1b3a12: Log is configured to *not* fsync() on all Append() calls I0507 06:33:29.939990 15164 log.cc:525] T 0712c795dc4648ffa156a17cf52d071c P 8c9db460feac4f3ea7d87157753cc05f: Log is configured to *not* fsync() on all Append() calls I0507 06:33:29.941572 15165 tablet_bootstrap.cc:439] T 0712c795dc4648ffa156a17cf52d071c P 9d7b3cf9bef345e38290f5f6ec1b3a12: No bootstrap required, opened a new log I0507 06:33:29.941813 15165 ts_tablet_manager.cc:1053] T 0712c795dc4648ffa156a17cf52d071c P 9d7b3cf9bef345e38290f5f6ec1b3a12: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.000s I0507 06:33:29.941951 15164 tablet_bootstrap.cc:439] T 0712c795dc4648ffa156a17cf52d071c P 8c9db460feac4f3ea7d87157753cc05f: No bootstrap required, opened a new log I0507 06:33:29.942252 15164 ts_tablet_manager.cc:1053] T 0712c795dc4648ffa156a17cf52d071c P 8c9db460feac4f3ea7d87157753cc05f: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.004s I0507 06:33:29.945214 15168 tablet_bootstrap.cc:439] T 0712c795dc4648ffa156a17cf52d071c P 69ac1841085d4196ac900d65a0238f79: Bootstrap starting. I0507 06:33:29.947533 15168 tablet_bootstrap.cc:590] T 0712c795dc4648ffa156a17cf52d071c P 69ac1841085d4196ac900d65a0238f79: No blocks or log segments found. Creating new log. I0507 06:33:29.948416 15168 log.cc:525] T 0712c795dc4648ffa156a17cf52d071c P 69ac1841085d4196ac900d65a0238f79: Log is configured to *not* fsync() on all Append() calls I0507 06:33:29.948734 15164 raft_consensus.cc:340] T 0712c795dc4648ffa156a17cf52d071c P 8c9db460feac4f3ea7d87157753cc05f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d7b3cf9bef345e38290f5f6ec1b3a12" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 38001 } } peers { permanent_uuid: "69ac1841085d4196ac900d65a0238f79" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 43169 } } peers { permanent_uuid: "8c9db460feac4f3ea7d87157753cc05f" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 38149 } } I0507 06:33:29.949134 15164 raft_consensus.cc:366] T 0712c795dc4648ffa156a17cf52d071c P 8c9db460feac4f3ea7d87157753cc05f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0507 06:33:29.949194 15164 raft_consensus.cc:705] T 0712c795dc4648ffa156a17cf52d071c P 8c9db460feac4f3ea7d87157753cc05f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c9db460feac4f3ea7d87157753cc05f, State: Initialized, Role: FOLLOWER I0507 06:33:29.949681 15164 consensus_queue.cc:229] T 0712c795dc4648ffa156a17cf52d071c P 8c9db460feac4f3ea7d87157753cc05f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d7b3cf9bef345e38290f5f6ec1b3a12" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 38001 } } peers { permanent_uuid: "69ac1841085d4196ac900d65a0238f79" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 43169 } } peers { permanent_uuid: "8c9db460feac4f3ea7d87157753cc05f" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 38149 } } I0507 06:33:29.951460 15165 raft_consensus.cc:340] T 0712c795dc4648ffa156a17cf52d071c P 9d7b3cf9bef345e38290f5f6ec1b3a12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d7b3cf9bef345e38290f5f6ec1b3a12" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 38001 } } peers { permanent_uuid: "69ac1841085d4196ac900d65a0238f79" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 43169 } } peers { permanent_uuid: "8c9db460feac4f3ea7d87157753cc05f" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 38149 } } I0507 06:33:29.951859 15165 raft_consensus.cc:366] T 0712c795dc4648ffa156a17cf52d071c P 9d7b3cf9bef345e38290f5f6ec1b3a12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0507 06:33:29.951915 15165 raft_consensus.cc:705] T 0712c795dc4648ffa156a17cf52d071c P 9d7b3cf9bef345e38290f5f6ec1b3a12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d7b3cf9bef345e38290f5f6ec1b3a12, State: Initialized, Role: FOLLOWER I0507 06:33:29.952353 15165 consensus_queue.cc:229] T 0712c795dc4648ffa156a17cf52d071c P 9d7b3cf9bef345e38290f5f6ec1b3a12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d7b3cf9bef345e38290f5f6ec1b3a12" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 38001 } } peers { permanent_uuid: "69ac1841085d4196ac900d65a0238f79" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 43169 } } peers { permanent_uuid: "8c9db460feac4f3ea7d87157753cc05f" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 38149 } } I0507 06:33:29.953007 15145 heartbeater.cc:467] Master 127.0.57.254:43845 was elected leader, sending a full tablet report... I0507 06:33:29.953478 15164 ts_tablet_manager.cc:1081] T 0712c795dc4648ffa156a17cf52d071c P 8c9db460feac4f3ea7d87157753cc05f: Time spent starting tablet: real 0.011s user 0.009s sys 0.002s I0507 06:33:29.954012 15023 heartbeater.cc:467] Master 127.0.57.254:43845 was elected leader, sending a full tablet report... I0507 06:33:29.954434 15165 ts_tablet_manager.cc:1081] T 0712c795dc4648ffa156a17cf52d071c P 9d7b3cf9bef345e38290f5f6ec1b3a12: Time spent starting tablet: real 0.013s user 0.008s sys 0.000s I0507 06:33:29.954963 15168 tablet_bootstrap.cc:439] T 0712c795dc4648ffa156a17cf52d071c P 69ac1841085d4196ac900d65a0238f79: No bootstrap required, opened a new log I0507 06:33:29.955215 15168 ts_tablet_manager.cc:1053] T 0712c795dc4648ffa156a17cf52d071c P 69ac1841085d4196ac900d65a0238f79: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.000s W0507 06:33:29.956363 15146 tablet.cc:1786] T 0712c795dc4648ffa156a17cf52d071c P 8c9db460feac4f3ea7d87157753cc05f: Can't schedule compaction. Clean time has not been advanced past its initial value. I0507 06:33:29.960569 15168 raft_consensus.cc:340] T 0712c795dc4648ffa156a17cf52d071c P 69ac1841085d4196ac900d65a0238f79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d7b3cf9bef345e38290f5f6ec1b3a12" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 38001 } } peers { permanent_uuid: "69ac1841085d4196ac900d65a0238f79" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 43169 } } peers { permanent_uuid: "8c9db460feac4f3ea7d87157753cc05f" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 38149 } } I0507 06:33:29.960911 15168 raft_consensus.cc:366] T 0712c795dc4648ffa156a17cf52d071c P 69ac1841085d4196ac900d65a0238f79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0507 06:33:29.960990 15168 raft_consensus.cc:705] T 0712c795dc4648ffa156a17cf52d071c P 69ac1841085d4196ac900d65a0238f79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69ac1841085d4196ac900d65a0238f79, State: Initialized, Role: FOLLOWER I0507 06:33:29.961416 15168 consensus_queue.cc:229] T 0712c795dc4648ffa156a17cf52d071c P 69ac1841085d4196ac900d65a0238f79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d7b3cf9bef345e38290f5f6ec1b3a12" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 38001 } } peers { permanent_uuid: "69ac1841085d4196ac900d65a0238f79" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 43169 } } peers { permanent_uuid: "8c9db460feac4f3ea7d87157753cc05f" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 38149 } } I0507 06:33:29.962293 14900 heartbeater.cc:467] Master 127.0.57.254:43845 was elected leader, sending a full tablet report... I0507 06:33:29.962576 15168 ts_tablet_manager.cc:1081] T 0712c795dc4648ffa156a17cf52d071c P 69ac1841085d4196ac900d65a0238f79: Time spent starting tablet: real 0.007s user 0.003s sys 0.004s W0507 06:33:29.981376 15024 tablet.cc:1786] T 0712c795dc4648ffa156a17cf52d071c P 9d7b3cf9bef345e38290f5f6ec1b3a12: Can't schedule compaction. Clean time has not been advanced past its initial value. I0507 06:33:29.984289 15169 raft_consensus.cc:472] T 0712c795dc4648ffa156a17cf52d071c P 8c9db460feac4f3ea7d87157753cc05f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0507 06:33:29.984541 15169 raft_consensus.cc:494] T 0712c795dc4648ffa156a17cf52d071c P 8c9db460feac4f3ea7d87157753cc05f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d7b3cf9bef345e38290f5f6ec1b3a12" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 38001 } } peers { permanent_uuid: "69ac1841085d4196ac900d65a0238f79" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 43169 } } peers { permanent_uuid: "8c9db460feac4f3ea7d87157753cc05f" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 38149 } } I0507 06:33:29.985865 15169 leader_election.cc:296] T 0712c795dc4648ffa156a17cf52d071c P 8c9db460feac4f3ea7d87157753cc05f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9d7b3cf9bef345e38290f5f6ec1b3a12 (127.0.57.194:38001), 69ac1841085d4196ac900d65a0238f79 (127.0.57.193:43169) I0507 06:33:29.991117 14857 tablet_service.cc:1286] Received RequestConsensusVote() RPC: tablet_id: "0712c795dc4648ffa156a17cf52d071c" candidate_uuid: "8c9db460feac4f3ea7d87157753cc05f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69ac1841085d4196ac900d65a0238f79" is_pre_election: true I0507 06:33:29.991461 14857 raft_consensus.cc:2346] T 0712c795dc4648ffa156a17cf52d071c P 69ac1841085d4196ac900d65a0238f79 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c9db460feac4f3ea7d87157753cc05f in term 0. I0507 06:33:29.991824 14980 tablet_service.cc:1286] Received RequestConsensusVote() RPC: tablet_id: "0712c795dc4648ffa156a17cf52d071c" candidate_uuid: "8c9db460feac4f3ea7d87157753cc05f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d7b3cf9bef345e38290f5f6ec1b3a12" is_pre_election: true I0507 06:33:29.992080 15035 leader_election.cc:310] T 0712c795dc4648ffa156a17cf52d071c P 8c9db460feac4f3ea7d87157753cc05f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 69ac1841085d4196ac900d65a0238f79, 8c9db460feac4f3ea7d87157753cc05f; no voters: I0507 06:33:29.992130 14980 raft_consensus.cc:2346] T 0712c795dc4648ffa156a17cf52d071c P 9d7b3cf9bef345e38290f5f6ec1b3a12 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c9db460feac4f3ea7d87157753cc05f in term 0. I0507 06:33:29.992605 15169 raft_consensus.cc:2640] T 0712c795dc4648ffa156a17cf52d071c P 8c9db460feac4f3ea7d87157753cc05f [term 0 FOLLOWER]: Leader pre-election won for term 1 I0507 06:33:29.992733 15169 raft_consensus.cc:472] T 0712c795dc4648ffa156a17cf52d071c P 8c9db460feac4f3ea7d87157753cc05f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0507 06:33:29.992800 15169 raft_consensus.cc:2886] T 0712c795dc4648ffa156a17cf52d071c P 8c9db460feac4f3ea7d87157753cc05f [term 0 FOLLOWER]: Advancing to term 1 I0507 06:33:29.994917 15169 raft_consensus.cc:494] T 0712c795dc4648ffa156a17cf52d071c P 8c9db460feac4f3ea7d87157753cc05f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d7b3cf9bef345e38290f5f6ec1b3a12" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 38001 } } peers { permanent_uuid: "69ac1841085d4196ac900d65a0238f79" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 43169 } } peers { permanent_uuid: "8c9db460feac4f3ea7d87157753cc05f" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 38149 } } I0507 06:33:29.995543 15169 leader_election.cc:296] T 0712c795dc4648ffa156a17cf52d071c P 8c9db460feac4f3ea7d87157753cc05f [CANDIDATE]: Term 1 election: Requested vote from peers 9d7b3cf9bef345e38290f5f6ec1b3a12 (127.0.57.194:38001), 69ac1841085d4196ac900d65a0238f79 (127.0.57.193:43169) I0507 06:33:29.996032 14980 tablet_service.cc:1286] Received RequestConsensusVote() RPC: tablet_id: "0712c795dc4648ffa156a17cf52d071c" candidate_uuid: "8c9db460feac4f3ea7d87157753cc05f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d7b3cf9bef345e38290f5f6ec1b3a12" I0507 06:33:29.996098 14857 tablet_service.cc:1286] Received RequestConsensusVote() RPC: tablet_id: "0712c795dc4648ffa156a17cf52d071c" candidate_uuid: "8c9db460feac4f3ea7d87157753cc05f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69ac1841085d4196ac900d65a0238f79" I0507 06:33:29.996204 14980 raft_consensus.cc:2886] T 0712c795dc4648ffa156a17cf52d071c P 9d7b3cf9bef345e38290f5f6ec1b3a12 [term 0 FOLLOWER]: Advancing to term 1 I0507 06:33:29.996243 14857 raft_consensus.cc:2886] T 0712c795dc4648ffa156a17cf52d071c P 69ac1841085d4196ac900d65a0238f79 [term 0 FOLLOWER]: Advancing to term 1 W0507 06:33:29.997107 14901 tablet.cc:1786] T 0712c795dc4648ffa156a17cf52d071c P 69ac1841085d4196ac900d65a0238f79: Can't schedule compaction. Clean time has not been advanced past its initial value. I0507 06:33:29.998054 14980 raft_consensus.cc:2346] T 0712c795dc4648ffa156a17cf52d071c P 9d7b3cf9bef345e38290f5f6ec1b3a12 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c9db460feac4f3ea7d87157753cc05f in term 1. I0507 06:33:29.998284 14857 raft_consensus.cc:2346] T 0712c795dc4648ffa156a17cf52d071c P 69ac1841085d4196ac900d65a0238f79 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c9db460feac4f3ea7d87157753cc05f in term 1. I0507 06:33:29.998497 15033 leader_election.cc:310] T 0712c795dc4648ffa156a17cf52d071c P 8c9db460feac4f3ea7d87157753cc05f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c9db460feac4f3ea7d87157753cc05f, 9d7b3cf9bef345e38290f5f6ec1b3a12; no voters: I0507 06:33:29.998813 15169 raft_consensus.cc:2640] T 0712c795dc4648ffa156a17cf52d071c P 8c9db460feac4f3ea7d87157753cc05f [term 1 FOLLOWER]: Leader election won for term 1 I0507 06:33:29.999236 15169 raft_consensus.cc:667] T 0712c795dc4648ffa156a17cf52d071c P 8c9db460feac4f3ea7d87157753cc05f [term 1 LEADER]: Becoming Leader. State: Replica: 8c9db460feac4f3ea7d87157753cc05f, State: Running, Role: LEADER I0507 06:33:29.999661 15169 consensus_queue.cc:206] T 0712c795dc4648ffa156a17cf52d071c P 8c9db460feac4f3ea7d87157753cc05f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d7b3cf9bef345e38290f5f6ec1b3a12" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 38001 } } peers { permanent_uuid: "69ac1841085d4196ac900d65a0238f79" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 43169 } } peers { permanent_uuid: "8c9db460feac4f3ea7d87157753cc05f" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 38149 } } I0507 06:33:30.003823 14735 catalog_manager.cc:3988] T 0712c795dc4648ffa156a17cf52d071c P 8c9db460feac4f3ea7d87157753cc05f reported cstate change: term changed from 0 to 1, leader changed from to 8c9db460feac4f3ea7d87157753cc05f (127.0.57.195). New cstate: current_term: 1 leader_uuid: "8c9db460feac4f3ea7d87157753cc05f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d7b3cf9bef345e38290f5f6ec1b3a12" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 38001 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "69ac1841085d4196ac900d65a0238f79" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 43169 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c9db460feac4f3ea7d87157753cc05f" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 38149 } health_report { overall_health: HEALTHY } } } I0507 06:33:30.019929 231 external_mini_cluster.cc:623] 3 TS(s) registered with all masters I0507 06:33:30.021591 231 ts_itest-base.cc:232] Waiting for 1 tablets on tserver 69ac1841085d4196ac900d65a0238f79 to finish bootstrapping I0507 06:33:30.028365 231 ts_itest-base.cc:232] Waiting for 1 tablets on tserver 9d7b3cf9bef345e38290f5f6ec1b3a12 to finish bootstrapping I0507 06:33:30.035012 231 ts_itest-base.cc:232] Waiting for 1 tablets on tserver 8c9db460feac4f3ea7d87157753cc05f to finish bootstrapping I0507 06:33:30.062594 231 cluster_itest_util.cc:258] Not converged past 1 yet: 0.0 0.0 1.1 I0507 06:33:30.167477 231 cluster_itest_util.cc:258] Not converged past 1 yet: 0.0 0.0 1.1 I0507 06:33:30.372139 231 cluster_itest_util.cc:258] Not converged past 1 yet: 0.0 0.0 1.1 I0507 06:33:30.514273 14857 raft_consensus.cc:1181] T 0712c795dc4648ffa156a17cf52d071c P 69ac1841085d4196ac900d65a0238f79 [term 1 FOLLOWER]: Refusing update from remote peer 8c9db460feac4f3ea7d87157753cc05f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch) I0507 06:33:30.515203 15177 consensus_queue.cc:984] T 0712c795dc4648ffa156a17cf52d071c P 8c9db460feac4f3ea7d87157753cc05f [LEADER]: Connected to new peer: Peer: permanent_uuid: "69ac1841085d4196ac900d65a0238f79" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 43169 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0507 06:33:30.522672 14980 raft_consensus.cc:1181] T 0712c795dc4648ffa156a17cf52d071c P 9d7b3cf9bef345e38290f5f6ec1b3a12 [term 1 FOLLOWER]: Refusing update from remote peer 8c9db460feac4f3ea7d87157753cc05f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch) I0507 06:33:30.523334 15177 consensus_queue.cc:984] T 0712c795dc4648ffa156a17cf52d071c P 8c9db460feac4f3ea7d87157753cc05f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d7b3cf9bef345e38290f5f6ec1b3a12" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 38001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0507 06:33:30.676805 231 external_mini_cluster.cc:1172] Attempting to check leaks for /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver pid 15026 I0507 06:33:30.677809 15142 generic_service.cc:171] Checking for leaks (request via RPC) I0507 06:33:30.727792 231 external_mini_cluster.cc:1131] Killing /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver with pid 15026 I0507 06:33:32.022702 15189 raft_consensus.cc:472] T 0712c795dc4648ffa156a17cf52d071c P 69ac1841085d4196ac900d65a0238f79 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8c9db460feac4f3ea7d87157753cc05f) I0507 06:33:32.022903 15189 raft_consensus.cc:494] T 0712c795dc4648ffa156a17cf52d071c P 69ac1841085d4196ac900d65a0238f79 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d7b3cf9bef345e38290f5f6ec1b3a12" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 38001 } } peers { permanent_uuid: "69ac1841085d4196ac900d65a0238f79" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 43169 } } peers { permanent_uuid: "8c9db460feac4f3ea7d87157753cc05f" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 38149 } } I0507 06:33:32.024314 15189 leader_election.cc:296] T 0712c795dc4648ffa156a17cf52d071c P 69ac1841085d4196ac900d65a0238f79 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9d7b3cf9bef345e38290f5f6ec1b3a12 (127.0.57.194:38001), 8c9db460feac4f3ea7d87157753cc05f (127.0.57.195:38149) I0507 06:33:32.029677 15192 raft_consensus.cc:472] T 0712c795dc4648ffa156a17cf52d071c P 9d7b3cf9bef345e38290f5f6ec1b3a12 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8c9db460feac4f3ea7d87157753cc05f) I0507 06:33:32.030012 15192 raft_consensus.cc:494] T 0712c795dc4648ffa156a17cf52d071c P 9d7b3cf9bef345e38290f5f6ec1b3a12 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d7b3cf9bef345e38290f5f6ec1b3a12" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 38001 } } peers { permanent_uuid: "69ac1841085d4196ac900d65a0238f79" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 43169 } } peers { permanent_uuid: "8c9db460feac4f3ea7d87157753cc05f" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 38149 } } I0507 06:33:32.031108 15192 leader_election.cc:296] T 0712c795dc4648ffa156a17cf52d071c P 9d7b3cf9bef345e38290f5f6ec1b3a12 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 69ac1841085d4196ac900d65a0238f79 (127.0.57.193:43169), 8c9db460feac4f3ea7d87157753cc05f (127.0.57.195:38149) W0507 06:33:32.032182 14790 leader_election.cc:341] T 0712c795dc4648ffa156a17cf52d071c P 69ac1841085d4196ac900d65a0238f79 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 8c9db460feac4f3ea7d87157753cc05f (127.0.57.195:38149): Network error: Client connection negotiation failed: client connection to 127.0.57.195:38149: connect: Connection refused (error 111) W0507 06:33:32.033731 14913 leader_election.cc:341] T 0712c795dc4648ffa156a17cf52d071c P 9d7b3cf9bef345e38290f5f6ec1b3a12 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 8c9db460feac4f3ea7d87157753cc05f (127.0.57.195:38149): Network error: Client connection negotiation failed: client connection to 127.0.57.195:38149: connect: Connection refused (error 111) I0507 06:33:32.037168 14980 tablet_service.cc:1286] Received RequestConsensusVote() RPC: tablet_id: "0712c795dc4648ffa156a17cf52d071c" candidate_uuid: "69ac1841085d4196ac900d65a0238f79" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9d7b3cf9bef345e38290f5f6ec1b3a12" is_pre_election: true I0507 06:33:32.037391 14980 raft_consensus.cc:2346] T 0712c795dc4648ffa156a17cf52d071c P 9d7b3cf9bef345e38290f5f6ec1b3a12 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69ac1841085d4196ac900d65a0238f79 in term 1. I0507 06:33:32.037933 14788 leader_election.cc:310] T 0712c795dc4648ffa156a17cf52d071c P 69ac1841085d4196ac900d65a0238f79 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 69ac1841085d4196ac900d65a0238f79, 9d7b3cf9bef345e38290f5f6ec1b3a12; no voters: 8c9db460feac4f3ea7d87157753cc05f I0507 06:33:32.038307 14857 tablet_service.cc:1286] Received RequestConsensusVote() RPC: tablet_id: "0712c795dc4648ffa156a17cf52d071c" candidate_uuid: "9d7b3cf9bef345e38290f5f6ec1b3a12" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "69ac1841085d4196ac900d65a0238f79" is_pre_election: true I0507 06:33:32.038501 15189 raft_consensus.cc:2640] T 0712c795dc4648ffa156a17cf52d071c P 69ac1841085d4196ac900d65a0238f79 [term 1 FOLLOWER]: Leader pre-election won for term 2 I0507 06:33:32.038625 15189 raft_consensus.cc:472] T 0712c795dc4648ffa156a17cf52d071c P 69ac1841085d4196ac900d65a0238f79 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 8c9db460feac4f3ea7d87157753cc05f) I0507 06:33:32.038683 15189 raft_consensus.cc:2886] T 0712c795dc4648ffa156a17cf52d071c P 69ac1841085d4196ac900d65a0238f79 [term 1 FOLLOWER]: Advancing to term 2 I0507 06:33:32.040437 15189 raft_consensus.cc:494] T 0712c795dc4648ffa156a17cf52d071c P 69ac1841085d4196ac900d65a0238f79 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d7b3cf9bef345e38290f5f6ec1b3a12" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 38001 } } peers { permanent_uuid: "69ac1841085d4196ac900d65a0238f79" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 43169 } } peers { permanent_uuid: "8c9db460feac4f3ea7d87157753cc05f" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 38149 } } I0507 06:33:32.040855 14857 raft_consensus.cc:2276] T 0712c795dc4648ffa156a17cf52d071c P 69ac1841085d4196ac900d65a0238f79 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9d7b3cf9bef345e38290f5f6ec1b3a12 in current term 2: Already voted for candidate 69ac1841085d4196ac900d65a0238f79 in this term. I0507 06:33:32.041081 15189 leader_election.cc:296] T 0712c795dc4648ffa156a17cf52d071c P 69ac1841085d4196ac900d65a0238f79 [CANDIDATE]: Term 2 election: Requested vote from peers 9d7b3cf9bef345e38290f5f6ec1b3a12 (127.0.57.194:38001), 8c9db460feac4f3ea7d87157753cc05f (127.0.57.195:38149) I0507 06:33:32.041638 14980 tablet_service.cc:1286] Received RequestConsensusVote() RPC: tablet_id: "0712c795dc4648ffa156a17cf52d071c" candidate_uuid: "69ac1841085d4196ac900d65a0238f79" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9d7b3cf9bef345e38290f5f6ec1b3a12" W0507 06:33:32.041731 14790 leader_election.cc:341] T 0712c795dc4648ffa156a17cf52d071c P 69ac1841085d4196ac900d65a0238f79 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 8c9db460feac4f3ea7d87157753cc05f (127.0.57.195:38149): Network error: Client connection negotiation failed: client connection to 127.0.57.195:38149: connect: Connection refused (error 111) I0507 06:33:32.041822 14980 raft_consensus.cc:2886] T 0712c795dc4648ffa156a17cf52d071c P 9d7b3cf9bef345e38290f5f6ec1b3a12 [term 1 FOLLOWER]: Advancing to term 2 I0507 06:33:32.043498 14980 raft_consensus.cc:2346] T 0712c795dc4648ffa156a17cf52d071c P 9d7b3cf9bef345e38290f5f6ec1b3a12 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69ac1841085d4196ac900d65a0238f79 in term 2. I0507 06:33:32.043915 14913 leader_election.cc:310] T 0712c795dc4648ffa156a17cf52d071c P 9d7b3cf9bef345e38290f5f6ec1b3a12 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9d7b3cf9bef345e38290f5f6ec1b3a12; no voters: 69ac1841085d4196ac900d65a0238f79, 8c9db460feac4f3ea7d87157753cc05f I0507 06:33:32.044399 15192 raft_consensus.cc:2588] T 0712c795dc4648ffa156a17cf52d071c P 9d7b3cf9bef345e38290f5f6ec1b3a12 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority I0507 06:33:32.044651 14788 leader_election.cc:310] T 0712c795dc4648ffa156a17cf52d071c P 69ac1841085d4196ac900d65a0238f79 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 69ac1841085d4196ac900d65a0238f79, 9d7b3cf9bef345e38290f5f6ec1b3a12; no voters: 8c9db460feac4f3ea7d87157753cc05f I0507 06:33:32.044975 15189 raft_consensus.cc:2640] T 0712c795dc4648ffa156a17cf52d071c P 69ac1841085d4196ac900d65a0238f79 [term 2 FOLLOWER]: Leader election won for term 2 I0507 06:33:32.045336 15189 raft_consensus.cc:667] T 0712c795dc4648ffa156a17cf52d071c P 69ac1841085d4196ac900d65a0238f79 [term 2 LEADER]: Becoming Leader. State: Replica: 69ac1841085d4196ac900d65a0238f79, State: Running, Role: LEADER I0507 06:33:32.045694 15189 consensus_queue.cc:206] T 0712c795dc4648ffa156a17cf52d071c P 69ac1841085d4196ac900d65a0238f79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d7b3cf9bef345e38290f5f6ec1b3a12" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 38001 } } peers { permanent_uuid: "69ac1841085d4196ac900d65a0238f79" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 43169 } } peers { permanent_uuid: "8c9db460feac4f3ea7d87157753cc05f" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 38149 } } I0507 06:33:32.049450 14735 catalog_manager.cc:3988] T 0712c795dc4648ffa156a17cf52d071c P 69ac1841085d4196ac900d65a0238f79 reported cstate change: term changed from 1 to 2, leader changed from 8c9db460feac4f3ea7d87157753cc05f (127.0.57.195) to 69ac1841085d4196ac900d65a0238f79 (127.0.57.193). New cstate: current_term: 2 leader_uuid: "69ac1841085d4196ac900d65a0238f79" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d7b3cf9bef345e38290f5f6ec1b3a12" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 38001 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "69ac1841085d4196ac900d65a0238f79" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 43169 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8c9db460feac4f3ea7d87157753cc05f" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 38149 } health_report { overall_health: UNKNOWN } } } I0507 06:33:32.512025 14980 raft_consensus.cc:1181] T 0712c795dc4648ffa156a17cf52d071c P 9d7b3cf9bef345e38290f5f6ec1b3a12 [term 2 FOLLOWER]: Refusing update from remote peer 69ac1841085d4196ac900d65a0238f79: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch) I0507 06:33:32.512979 15189 consensus_queue.cc:984] T 0712c795dc4648ffa156a17cf52d071c P 69ac1841085d4196ac900d65a0238f79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d7b3cf9bef345e38290f5f6ec1b3a12" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 38001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s W0507 06:33:32.518431 14790 consensus_peers.cc:457] T 0712c795dc4648ffa156a17cf52d071c P 69ac1841085d4196ac900d65a0238f79 -> Peer 8c9db460feac4f3ea7d87157753cc05f (127.0.57.195:38149): Couldn't send request to peer 8c9db460feac4f3ea7d87157753cc05f. Status: Network error: Client connection negotiation failed: client connection to 127.0.57.195:38149: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry. W0507 06:33:35.056730 14790 consensus_peers.cc:457] T 0712c795dc4648ffa156a17cf52d071c P 69ac1841085d4196ac900d65a0238f79 -> Peer 8c9db460feac4f3ea7d87157753cc05f (127.0.57.195:38149): Couldn't send request to peer 8c9db460feac4f3ea7d87157753cc05f. Status: Network error: Client connection negotiation failed: client connection to 127.0.57.195:38149: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry. I0507 06:33:37.077491 15206 consensus_queue.cc:544] T 0712c795dc4648ffa156a17cf52d071c P 69ac1841085d4196ac900d65a0238f79 [LEADER]: Leader has been unable to successfully communicate with peer 8c9db460feac4f3ea7d87157753cc05f for more than 5 seconds (5.031s) W0507 06:33:37.679260 14790 consensus_peers.cc:457] T 0712c795dc4648ffa156a17cf52d071c P 69ac1841085d4196ac900d65a0238f79 -> Peer 8c9db460feac4f3ea7d87157753cc05f (127.0.57.195:38149): Couldn't send request to peer 8c9db460feac4f3ea7d87157753cc05f. Status: Network error: Client connection negotiation failed: client connection to 127.0.57.195:38149: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry. W0507 06:33:40.317093 14790 consensus_peers.cc:457] T 0712c795dc4648ffa156a17cf52d071c P 69ac1841085d4196ac900d65a0238f79 -> Peer 8c9db460feac4f3ea7d87157753cc05f (127.0.57.195:38149): Couldn't send request to peer 8c9db460feac4f3ea7d87157753cc05f. Status: Network error: Client connection negotiation failed: client connection to 127.0.57.195:38149: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry. W0507 06:33:42.459558 14790 consensus_peers.cc:457] T 0712c795dc4648ffa156a17cf52d071c P 69ac1841085d4196ac900d65a0238f79 -> Peer 8c9db460feac4f3ea7d87157753cc05f (127.0.57.195:38149): Couldn't send request to peer 8c9db460feac4f3ea7d87157753cc05f. Status: Network error: Client connection negotiation failed: client connection to 127.0.57.195:38149: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry. W0507 06:33:44.602902 14790 consensus_peers.cc:457] T 0712c795dc4648ffa156a17cf52d071c P 69ac1841085d4196ac900d65a0238f79 -> Peer 8c9db460feac4f3ea7d87157753cc05f (127.0.57.195:38149): Couldn't send request to peer 8c9db460feac4f3ea7d87157753cc05f. Status: Network error: Client connection negotiation failed: client connection to 127.0.57.195:38149: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry. W0507 06:33:46.914677 14790 consensus_peers.cc:457] T 0712c795dc4648ffa156a17cf52d071c P 69ac1841085d4196ac900d65a0238f79 -> Peer 8c9db460feac4f3ea7d87157753cc05f (127.0.57.195:38149): Couldn't send request to peer 8c9db460feac4f3ea7d87157753cc05f. Status: Network error: Client connection negotiation failed: client connection to 127.0.57.195:38149: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry. I0507 06:33:47.280503 231 test_util.cc:212] Using random seed: 1714638945 I0507 06:33:47.298624 231 test_workload.cc:306] TestWorkload: Skipping table creation because table TestTable already exists I0507 06:33:47.301482 231 raft_consensus-itest-base.cc:271] Waiting until we've written at least 4MB... I0507 06:33:47.447657 15024 maintenance_manager.cc:306] P 9d7b3cf9bef345e38290f5f6ec1b3a12: Scheduling FlushMRSOp(0712c795dc4648ffa156a17cf52d071c): 4207507 bytes log retention I0507 06:33:47.461334 14901 maintenance_manager.cc:306] P 69ac1841085d4196ac900d65a0238f79: Scheduling FlushMRSOp(0712c795dc4648ffa156a17cf52d071c): 3681626 bytes log retention I0507 06:33:47.467754 15236 consensus_queue.cc:735] T 0712c795dc4648ffa156a17cf52d071c P 69ac1841085d4196ac900d65a0238f79 [LEADER]: Peer 8c9db460feac4f3ea7d87157753cc05f is lagging by at least 27 ops behind the committed index I0507 06:33:47.486430 231 raft_consensus-itest-base.cc:277] Waiting for log GC on 69ac1841085d4196ac900d65a0238f79 I0507 06:33:47.534620 14918 maintenance_manager.cc:501] P 9d7b3cf9bef345e38290f5f6ec1b3a12: FlushMRSOp(0712c795dc4648ffa156a17cf52d071c) complete. Timing: real 0.086s user 0.062s sys 0.008s Metrics: {"bytes_written":136001,"cfile_init":1,"compiler_manager_pool.queue_time_us":4548,"data dirs.queue_time_us":2874,"data dirs.run_cpu_time_us":246,"data dirs.run_wall_time_us":3675,"drs_written":1,"lbm_read_time_us":116,"lbm_reads_lt_1ms":4,"lbm_write_time_us":700,"lbm_writes_lt_1ms":23,"rows_written":34,"thread_start_us":9360,"threads_started":2} I0507 06:33:47.536182 15024 maintenance_manager.cc:306] P 9d7b3cf9bef345e38290f5f6ec1b3a12: Scheduling LogGCOp(0712c795dc4648ffa156a17cf52d071c): free 2498236 bytes of WAL I0507 06:33:47.536666 14918 log_reader.cc:377] T 0712c795dc4648ffa156a17cf52d071c: removed 3 log segments from log reader I0507 06:33:47.536794 14918 log.cc:863] T 0712c795dc4648ffa156a17cf52d071c P 9d7b3cf9bef345e38290f5f6ec1b3a12: Deleting log segment in path: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_3.1557210500371863-231/raft_consensus-itest-cluster/ts-1/wal/wals/0712c795dc4648ffa156a17cf52d071c/wal-000000001 (ops 1-8) I0507 06:33:47.536947 14918 log.cc:863] T 0712c795dc4648ffa156a17cf52d071c P 9d7b3cf9bef345e38290f5f6ec1b3a12: Deleting log segment in path: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_3.1557210500371863-231/raft_consensus-itest-cluster/ts-1/wal/wals/0712c795dc4648ffa156a17cf52d071c/wal-000000002 (ops 9-15) I0507 06:33:47.537062 14918 log.cc:863] T 0712c795dc4648ffa156a17cf52d071c P 9d7b3cf9bef345e38290f5f6ec1b3a12: Deleting log segment in path: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_3.1557210500371863-231/raft_consensus-itest-cluster/ts-1/wal/wals/0712c795dc4648ffa156a17cf52d071c/wal-000000003 (ops 16-21) I0507 06:33:47.537739 14918 maintenance_manager.cc:501] P 9d7b3cf9bef345e38290f5f6ec1b3a12: LogGCOp(0712c795dc4648ffa156a17cf52d071c) complete. Timing: real 0.001s user 0.001s sys 0.000s Metrics: {} I0507 06:33:47.538126 15024 maintenance_manager.cc:306] P 9d7b3cf9bef345e38290f5f6ec1b3a12: Scheduling UndoDeltaBlockGCOp(0712c795dc4648ffa156a17cf52d071c): 516 bytes on disk I0507 06:33:47.538985 14918 maintenance_manager.cc:501] P 9d7b3cf9bef345e38290f5f6ec1b3a12: UndoDeltaBlockGCOp(0712c795dc4648ffa156a17cf52d071c) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":75,"lbm_reads_lt_1ms":4} I0507 06:33:47.544109 14795 maintenance_manager.cc:501] P 69ac1841085d4196ac900d65a0238f79: FlushMRSOp(0712c795dc4648ffa156a17cf52d071c) complete. Timing: real 0.082s user 0.051s sys 0.022s Metrics: {"bytes_written":136001,"cfile_init":1,"compiler_manager_pool.queue_time_us":7601,"data dirs.queue_time_us":1150,"data dirs.run_cpu_time_us":189,"data dirs.run_wall_time_us":2133,"drs_written":1,"lbm_read_time_us":72,"lbm_reads_lt_1ms":4,"lbm_write_time_us":542,"lbm_writes_lt_1ms":23,"rows_written":34,"thread_start_us":8357,"threads_started":2} I0507 06:33:47.545446 14901 maintenance_manager.cc:306] P 69ac1841085d4196ac900d65a0238f79: Scheduling LogGCOp(0712c795dc4648ffa156a17cf52d071c): free 2761256 bytes of WAL I0507 06:33:47.545809 14795 log_reader.cc:377] T 0712c795dc4648ffa156a17cf52d071c: removed 3 log segments from log reader I0507 06:33:47.545900 14795 log.cc:863] T 0712c795dc4648ffa156a17cf52d071c P 69ac1841085d4196ac900d65a0238f79: Deleting log segment in path: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_3.1557210500371863-231/raft_consensus-itest-cluster/ts-0/wal/wals/0712c795dc4648ffa156a17cf52d071c/wal-000000001 (ops 1-9) I0507 06:33:47.545995 14795 log.cc:863] T 0712c795dc4648ffa156a17cf52d071c P 69ac1841085d4196ac900d65a0238f79: Deleting log segment in path: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_3.1557210500371863-231/raft_consensus-itest-cluster/ts-0/wal/wals/0712c795dc4648ffa156a17cf52d071c/wal-000000002 (ops 10-16) I0507 06:33:47.546083 14795 log.cc:863] T 0712c795dc4648ffa156a17cf52d071c P 69ac1841085d4196ac900d65a0238f79: Deleting log segment in path: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_3.1557210500371863-231/raft_consensus-itest-cluster/ts-0/wal/wals/0712c795dc4648ffa156a17cf52d071c/wal-000000003 (ops 17-23) I0507 06:33:47.546828 14795 maintenance_manager.cc:501] P 69ac1841085d4196ac900d65a0238f79: LogGCOp(0712c795dc4648ffa156a17cf52d071c) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {} I0507 06:33:47.547204 14901 maintenance_manager.cc:306] P 69ac1841085d4196ac900d65a0238f79: Scheduling UndoDeltaBlockGCOp(0712c795dc4648ffa156a17cf52d071c): 516 bytes on disk I0507 06:33:47.547978 14795 maintenance_manager.cc:501] P 69ac1841085d4196ac900d65a0238f79: UndoDeltaBlockGCOp(0712c795dc4648ffa156a17cf52d071c) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":70,"lbm_reads_lt_1ms":4} I0507 06:33:47.548238 231 raft_consensus-itest-base.cc:282] Log GC complete on 69ac1841085d4196ac900d65a0238f79 I0507 06:33:48.069607 15240 consensus_queue.cc:689] T 0712c795dc4648ffa156a17cf52d071c P 69ac1841085d4196ac900d65a0238f79 [LEADER]: The logs necessary to catch up peer 8c9db460feac4f3ea7d87157753cc05f have been garbage collected. The follower will never be able to catch up (Not found: Failed to read ops 3..31: Segment 1 which contained index 3 has been GCed) I0507 06:33:48.069749 15240 consensus_queue.cc:544] T 0712c795dc4648ffa156a17cf52d071c P 69ac1841085d4196ac900d65a0238f79 [LEADER]: The logs necessary to catch up peer 8c9db460feac4f3ea7d87157753cc05f have been garbage collected. The replica will never be able to catch up I0507 06:33:48.073071 14734 catalog_manager.cc:3739] Sending ChangeConfig:REMOVE_PEER on tablet 0712c795dc4648ffa156a17cf52d071c (attempt 1) I0507 06:33:48.075611 14857 consensus_queue.cc:206] T 0712c795dc4648ffa156a17cf52d071c P 69ac1841085d4196ac900d65a0238f79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 38, Committed index: 38, Last appended: 2.38, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 39 OBSOLETE_local: false peers { permanent_uuid: "9d7b3cf9bef345e38290f5f6ec1b3a12" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 38001 } } peers { permanent_uuid: "69ac1841085d4196ac900d65a0238f79" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 43169 } } I0507 06:33:48.077972 14979 raft_consensus.cc:1181] T 0712c795dc4648ffa156a17cf52d071c P 9d7b3cf9bef345e38290f5f6ec1b3a12 [term 2 FOLLOWER]: Refusing update from remote peer 69ac1841085d4196ac900d65a0238f79: Log matching property violated. Preceding OpId in replica: term: 2 index: 38. Preceding OpId from leader: term: 2 index: 39. (index mismatch) I0507 06:33:48.078781 15240 consensus_queue.cc:984] T 0712c795dc4648ffa156a17cf52d071c P 69ac1841085d4196ac900d65a0238f79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d7b3cf9bef345e38290f5f6ec1b3a12" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 38001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 39, Last known committed idx: 38, Time since last communication: 0.000s I0507 06:33:48.081794 15253 raft_consensus.cc:2788] T 0712c795dc4648ffa156a17cf52d071c P 69ac1841085d4196ac900d65a0238f79 [term 2 LEADER]: Committing config change with OpId 2.39: config changed from index -1 to 39, VOTER 8c9db460feac4f3ea7d87157753cc05f (127.0.57.195) evicted. New config: { opid_index: 39 OBSOLETE_local: false peers { permanent_uuid: "9d7b3cf9bef345e38290f5f6ec1b3a12" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 38001 } } peers { permanent_uuid: "69ac1841085d4196ac900d65a0238f79" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 43169 } } } I0507 06:33:48.082608 14979 raft_consensus.cc:2788] T 0712c795dc4648ffa156a17cf52d071c P 9d7b3cf9bef345e38290f5f6ec1b3a12 [term 2 FOLLOWER]: Committing config change with OpId 2.39: config changed from index -1 to 39, VOTER 8c9db460feac4f3ea7d87157753cc05f (127.0.57.195) evicted. New config: { opid_index: 39 OBSOLETE_local: false peers { permanent_uuid: "9d7b3cf9bef345e38290f5f6ec1b3a12" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 38001 } } peers { permanent_uuid: "69ac1841085d4196ac900d65a0238f79" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 43169 } } } I0507 06:33:48.085206 14720 catalog_manager.cc:3531] ChangeConfig:REMOVE_PEER RPC for tablet 0712c795dc4648ffa156a17cf52d071c with cas_config_opid_index -1: ChangeConfig:REMOVE_PEER succeeded (attempt 1) I0507 06:33:48.086196 14734 catalog_manager.cc:3988] T 0712c795dc4648ffa156a17cf52d071c P 69ac1841085d4196ac900d65a0238f79 reported cstate change: config changed from index -1 to 39, VOTER 8c9db460feac4f3ea7d87157753cc05f (127.0.57.195) evicted. New cstate: current_term: 2 leader_uuid: "69ac1841085d4196ac900d65a0238f79" committed_config { opid_index: 39 OBSOLETE_local: false peers { permanent_uuid: "9d7b3cf9bef345e38290f5f6ec1b3a12" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 38001 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "69ac1841085d4196ac900d65a0238f79" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 43169 } health_report { overall_health: HEALTHY } } } I0507 06:33:48.090649 14734 catalog_manager.cc:3390] Sending DeleteTablet(TABLET_DATA_TOMBSTONED) for tablet 0712c795dc4648ffa156a17cf52d071c on 8c9db460feac4f3ea7d87157753cc05f (127.0.57.195:38149) (TS 8c9db460feac4f3ea7d87157753cc05f not found in new config with opid_index 39) W0507 06:33:48.097558 14720 catalog_manager.cc:3132] TS 8c9db460feac4f3ea7d87157753cc05f (127.0.57.195:38149): Delete Tablet RPC failed for tablet 0712c795dc4648ffa156a17cf52d071c: Network error: Client connection negotiation failed: client connection to 127.0.57.195:38149: connect: Connection refused (error 111) I0507 06:33:48.097748 14720 catalog_manager.cc:3168] Scheduling retry of 0712c795dc4648ffa156a17cf52d071c Delete Tablet RPC for TS=8c9db460feac4f3ea7d87157753cc05f with a delay of 32 ms (attempt = 1) I0507 06:33:48.130995 14720 catalog_manager.cc:3390] Sending DeleteTablet(TABLET_DATA_TOMBSTONED) for tablet 0712c795dc4648ffa156a17cf52d071c on 8c9db460feac4f3ea7d87157753cc05f (127.0.57.195:38149) (TS 8c9db460feac4f3ea7d87157753cc05f not found in new config with opid_index 39) W0507 06:33:48.132179 14720 catalog_manager.cc:3132] TS 8c9db460feac4f3ea7d87157753cc05f (127.0.57.195:38149): Delete Tablet RPC failed for tablet 0712c795dc4648ffa156a17cf52d071c: Network error: Client connection negotiation failed: client connection to 127.0.57.195:38149: connect: Connection refused (error 111) I0507 06:33:48.132268 14720 catalog_manager.cc:3168] Scheduling retry of 0712c795dc4648ffa156a17cf52d071c Delete Tablet RPC for TS=8c9db460feac4f3ea7d87157753cc05f with a delay of 79 ms (attempt = 2) I0507 06:33:48.142565 15024 maintenance_manager.cc:306] P 9d7b3cf9bef345e38290f5f6ec1b3a12: Scheduling LogGCOp(0712c795dc4648ffa156a17cf52d071c): free 1709271 bytes of WAL I0507 06:33:48.143036 14918 log_reader.cc:377] T 0712c795dc4648ffa156a17cf52d071c: removed 2 log segments from log reader I0507 06:33:48.143111 14918 log.cc:863] T 0712c795dc4648ffa156a17cf52d071c P 9d7b3cf9bef345e38290f5f6ec1b3a12: Deleting log segment in path: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_3.1557210500371863-231/raft_consensus-itest-cluster/ts-1/wal/wals/0712c795dc4648ffa156a17cf52d071c/wal-000000004 (ops 22-27) I0507 06:33:48.143210 14918 log.cc:863] T 0712c795dc4648ffa156a17cf52d071c P 9d7b3cf9bef345e38290f5f6ec1b3a12: Deleting log segment in path: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_3.1557210500371863-231/raft_consensus-itest-cluster/ts-1/wal/wals/0712c795dc4648ffa156a17cf52d071c/wal-000000005 (ops 28-34) I0507 06:33:48.143707 14918 maintenance_manager.cc:501] P 9d7b3cf9bef345e38290f5f6ec1b3a12: LogGCOp(0712c795dc4648ffa156a17cf52d071c) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {} I0507 06:33:48.152210 14901 maintenance_manager.cc:306] P 69ac1841085d4196ac900d65a0238f79: Scheduling LogGCOp(0712c795dc4648ffa156a17cf52d071c): free 920370 bytes of WAL I0507 06:33:48.152772 14795 log_reader.cc:377] T 0712c795dc4648ffa156a17cf52d071c: removed 1 log segments from log reader I0507 06:33:48.152861 14795 log.cc:863] T 0712c795dc4648ffa156a17cf52d071c P 69ac1841085d4196ac900d65a0238f79: Deleting log segment in path: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_3.1557210500371863-231/raft_consensus-itest-cluster/ts-0/wal/wals/0712c795dc4648ffa156a17cf52d071c/wal-000000004 (ops 24-30) I0507 06:33:48.153184 14795 maintenance_manager.cc:501] P 69ac1841085d4196ac900d65a0238f79: LogGCOp(0712c795dc4648ffa156a17cf52d071c) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {} I0507 06:33:48.211885 14720 catalog_manager.cc:3390] Sending DeleteTablet(TABLET_DATA_TOMBSTONED) for tablet 0712c795dc4648ffa156a17cf52d071c on 8c9db460feac4f3ea7d87157753cc05f (127.0.57.195:38149) (TS 8c9db460feac4f3ea7d87157753cc05f not found in new config with opid_index 39) W0507 06:33:48.213044 14720 catalog_manager.cc:3132] TS 8c9db460feac4f3ea7d87157753cc05f (127.0.57.195:38149): Delete Tablet RPC failed for tablet 0712c795dc4648ffa156a17cf52d071c: Network error: Client connection negotiation failed: client connection to 127.0.57.195:38149: connect: Connection refused (error 111) I0507 06:33:48.213183 14720 catalog_manager.cc:3168] Scheduling retry of 0712c795dc4648ffa156a17cf52d071c Delete Tablet RPC for TS=8c9db460feac4f3ea7d87157753cc05f with a delay of 112 ms (attempt = 3) I0507 06:33:48.326176 14720 catalog_manager.cc:3390] Sending DeleteTablet(TABLET_DATA_TOMBSTONED) for tablet 0712c795dc4648ffa156a17cf52d071c on 8c9db460feac4f3ea7d87157753cc05f (127.0.57.195:38149) (TS 8c9db460feac4f3ea7d87157753cc05f not found in new config with opid_index 39) W0507 06:33:48.327318 14720 catalog_manager.cc:3132] TS 8c9db460feac4f3ea7d87157753cc05f (127.0.57.195:38149): Delete Tablet RPC failed for tablet 0712c795dc4648ffa156a17cf52d071c: Network error: Client connection negotiation failed: client connection to 127.0.57.195:38149: connect: Connection refused (error 111) I0507 06:33:48.327404 14720 catalog_manager.cc:3168] Scheduling retry of 0712c795dc4648ffa156a17cf52d071c Delete Tablet RPC for TS=8c9db460feac4f3ea7d87157753cc05f with a delay of 175 ms (attempt = 4) I0507 06:33:48.503387 14720 catalog_manager.cc:3390] Sending DeleteTablet(TABLET_DATA_TOMBSTONED) for tablet 0712c795dc4648ffa156a17cf52d071c on 8c9db460feac4f3ea7d87157753cc05f (127.0.57.195:38149) (TS 8c9db460feac4f3ea7d87157753cc05f not found in new config with opid_index 39) W0507 06:33:48.504554 14720 catalog_manager.cc:3132] TS 8c9db460feac4f3ea7d87157753cc05f (127.0.57.195:38149): Delete Tablet RPC failed for tablet 0712c795dc4648ffa156a17cf52d071c: Network error: Client connection negotiation failed: client connection to 127.0.57.195:38149: connect: Connection refused (error 111) I0507 06:33:48.504664 14720 catalog_manager.cc:3168] Scheduling retry of 0712c795dc4648ffa156a17cf52d071c Delete Tablet RPC for TS=8c9db460feac4f3ea7d87157753cc05f with a delay of 278 ms (attempt = 5) I0507 06:33:48.783169 14720 catalog_manager.cc:3390] Sending DeleteTablet(TABLET_DATA_TOMBSTONED) for tablet 0712c795dc4648ffa156a17cf52d071c on 8c9db460feac4f3ea7d87157753cc05f (127.0.57.195:38149) (TS 8c9db460feac4f3ea7d87157753cc05f not found in new config with opid_index 39) W0507 06:33:48.784354 14720 catalog_manager.cc:3132] TS 8c9db460feac4f3ea7d87157753cc05f (127.0.57.195:38149): Delete Tablet RPC failed for tablet 0712c795dc4648ffa156a17cf52d071c: Network error: Client connection negotiation failed: client connection to 127.0.57.195:38149: connect: Connection refused (error 111) I0507 06:33:48.784432 14720 catalog_manager.cc:3168] Scheduling retry of 0712c795dc4648ffa156a17cf52d071c Delete Tablet RPC for TS=8c9db460feac4f3ea7d87157753cc05f with a delay of 528 ms (attempt = 6) I0507 06:33:49.313699 14720 catalog_manager.cc:3390] Sending DeleteTablet(TABLET_DATA_TOMBSTONED) for tablet 0712c795dc4648ffa156a17cf52d071c on 8c9db460feac4f3ea7d87157753cc05f (127.0.57.195:38149) (TS 8c9db460feac4f3ea7d87157753cc05f not found in new config with opid_index 39) W0507 06:33:49.314920 14720 catalog_manager.cc:3132] TS 8c9db460feac4f3ea7d87157753cc05f (127.0.57.195:38149): Delete Tablet RPC failed for tablet 0712c795dc4648ffa156a17cf52d071c: Network error: Client connection negotiation failed: client connection to 127.0.57.195:38149: connect: Connection refused (error 111) I0507 06:33:49.315011 14720 catalog_manager.cc:3168] Scheduling retry of 0712c795dc4648ffa156a17cf52d071c Delete Tablet RPC for TS=8c9db460feac4f3ea7d87157753cc05f with a delay of 1060 ms (attempt = 7) I0507 06:33:49.550714 231 raft_consensus-itest-base.cc:298] Servers converged with original term 2 I0507 06:33:49.550781 231 raft_consensus-itest-base.cc:306] Restarting 8c9db460feac4f3ea7d87157753cc05f I0507 06:33:49.551302 231 external_mini_cluster.cc:933] Running /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_3.1557210500371863-231/raft_consensus-itest-cluster/ts-2/wal --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_3.1557210500371863-231/raft_consensus-itest-cluster/ts-2/data --block_manager=log --rpc_bind_addresses=127.0.57.195:38149 --local_ip_for_outbound_sockets=127.0.57.195 --webserver_port=36001 --webserver_interface=127.0.57.195 --tserver_master_addrs=127.0.57.254:43845 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_3.1557210500371863-231/raft_consensus-itest-cluster/ts-2/logs --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_3.1557210500371863-231/raft_consensus-itest-cluster/ts-2/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=true --follower_unavailable_considered_failed_sec=5 --log_compression_codec=none --log_cache_size_limit_mb=1 --log_segment_size_mb=1 --log_async_preallocate_segments=false --log_min_segments_to_retain=1 --log_max_segments_to_retain=3 --maintenance_manager_polling_interval_ms=100 --log_target_replay_size_mb=1 --max_cell_size_bytes=1000000 with env {} WARNING: Logging before InitGoogleLogging() is written to STDERR W0507 06:33:49.738361 15263 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0507 06:33:49.738631 15263 flags.cc:404] Enabled unsafe flag: --max_cell_size_bytes=1000000 W0507 06:33:49.738679 15263 flags.cc:404] Enabled unsafe flag: --never_fsync=true W0507 06:33:49.740991 15263 flags.cc:404] Enabled experimental flag: --log_compression_codec=none W0507 06:33:49.741061 15263 flags.cc:404] Enabled experimental flag: --log_max_segments_to_retain=3 W0507 06:33:49.741106 15263 flags.cc:404] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true W0507 06:33:49.741190 15263 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W0507 06:33:49.741307 15263 flags.cc:404] Enabled experimental flag: --log_target_replay_size_mb=1 W0507 06:33:49.741372 15263 flags.cc:404] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.57.195 I0507 06:33:49.746434 15263 tablet_server_main.cc:78] Tablet server non-default flags: --follower_unavailable_considered_failed_sec=5 --log_compression_codec=none --log_max_segments_to_retain=3 --log_cache_size_limit_mb=1 --log_async_preallocate_segments=false --log_segment_size_mb=1 --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_3.1557210500371863-231/raft_consensus-itest-cluster/ts-2/data --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_3.1557210500371863-231/raft_consensus-itest-cluster/ts-2/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.0.57.195:38149 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_3.1557210500371863-231/raft_consensus-itest-cluster/ts-2/data/info.pb --webserver_interface=127.0.57.195 --webserver_port=36001 --max_cell_size_bytes=1000000 --tserver_master_addrs=127.0.57.254:43845 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --log_target_replay_size_mb=1 --maintenance_manager_polling_interval_ms=100 --enable_minidumps=false --local_ip_for_outbound_sockets=127.0.57.195 --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_3.1557210500371863-231/raft_consensus-itest-cluster/ts-2/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.10.0-cdh6.x-SNAPSHOT revision 7d0ee26697926e87af9ef395008799577f6ab43f build type FASTDEBUG built by jenkins at 06 May 2019 22:44:21 PST on quasar-knmpfx-1.vpc.cloudera.com build id 322017 ASAN enabled I0507 06:33:49.755162 15263 system_ntp.cc:202] NTP initialized. Skew: 500ppm Current error: 189016us I0507 06:33:49.755949 15263 fs_manager.cc:263] Metadata directory not provided I0507 06:33:49.756028 15263 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0507 06:33:49.763152 15263 fs_manager.cc:399] Time spent opening directory manager: real 0.005s user 0.002s sys 0.002s I0507 06:33:49.763998 15263 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0507 06:33:49.764797 15263 file_cache.cc:466] Constructed file cache lbm with capacity 419430 I0507 06:33:49.768430 15263 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s I0507 06:33:49.768587 15263 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_3.1557210500371863-231/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_3.1557210500371863-231/raft_consensus-itest-cluster/ts-2/wal uuid: "8c9db460feac4f3ea7d87157753cc05f" format_stamp: "Formatted at 2019-05-07 06:33:29 on dist-test-slave-dist-test-slave-rqp6" I0507 06:33:49.769731 15263 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_3.1557210500371863-231/raft_consensus-itest-cluster/ts-2/wal metadata directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_3.1557210500371863-231/raft_consensus-itest-cluster/ts-2/wal 1 data directories: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_3.1557210500371863-231/raft_consensus-itest-cluster/ts-2/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0507 06:33:49.818080 15263 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0507 06:33:49.820142 15263 ts_tablet_manager.cc:344] Loading tablet metadata (0/1 complete) I0507 06:33:49.824721 15263 ts_tablet_manager.cc:356] Loaded tablet metadata (1 total tablets, 1 live tablets) I0507 06:33:49.824789 15263 ts_tablet_manager.cc:362] Registering tablets (0/1 complete) I0507 06:33:49.828045 15279 tablet_bootstrap.cc:439] T 0712c795dc4648ffa156a17cf52d071c P 8c9db460feac4f3ea7d87157753cc05f: Bootstrap starting. I0507 06:33:49.832638 15263 ts_tablet_manager.cc:376] Registered 1 tablets I0507 06:33:49.844967 15279 log.cc:525] T 0712c795dc4648ffa156a17cf52d071c P 8c9db460feac4f3ea7d87157753cc05f: Log is configured to *not* fsync() on all Append() calls I0507 06:33:49.851184 15279 tablet_bootstrap.cc:439] T 0712c795dc4648ffa156a17cf52d071c P 8c9db460feac4f3ea7d87157753cc05f: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I0507 06:33:49.851814 15279 tablet_bootstrap.cc:439] T 0712c795dc4648ffa156a17cf52d071c P 8c9db460feac4f3ea7d87157753cc05f: Bootstrap complete. I0507 06:33:49.852257 15279 ts_tablet_manager.cc:1053] T 0712c795dc4648ffa156a17cf52d071c P 8c9db460feac4f3ea7d87157753cc05f: Time spent bootstrapping tablet: real 0.024s user 0.012s sys 0.004s I0507 06:33:49.870743 15279 raft_consensus.cc:340] T 0712c795dc4648ffa156a17cf52d071c P 8c9db460feac4f3ea7d87157753cc05f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d7b3cf9bef345e38290f5f6ec1b3a12" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 38001 } } peers { permanent_uuid: "69ac1841085d4196ac900d65a0238f79" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 43169 } } peers { permanent_uuid: "8c9db460feac4f3ea7d87157753cc05f" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 38149 } } I0507 06:33:49.875435 15279 raft_consensus.cc:705] T 0712c795dc4648ffa156a17cf52d071c P 8c9db460feac4f3ea7d87157753cc05f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c9db460feac4f3ea7d87157753cc05f, State: Initialized, Role: FOLLOWER I0507 06:33:49.876194 15279 consensus_queue.cc:229] T 0712c795dc4648ffa156a17cf52d071c P 8c9db460feac4f3ea7d87157753cc05f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d7b3cf9bef345e38290f5f6ec1b3a12" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 38001 } } peers { permanent_uuid: "69ac1841085d4196ac900d65a0238f79" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 43169 } } peers { permanent_uuid: "8c9db460feac4f3ea7d87157753cc05f" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 38149 } } I0507 06:33:49.879148 15279 ts_tablet_manager.cc:1081] T 0712c795dc4648ffa156a17cf52d071c P 8c9db460feac4f3ea7d87157753cc05f: Time spent starting tablet: real 0.026s user 0.012s sys 0.012s I0507 06:33:49.880544 15263 rpc_server.cc:205] RPC server started. Bound to: 127.0.57.195:38149 I0507 06:33:49.881750 15263 webserver.cc:308] Webserver started at http://127.0.57.195:36001/ using document root and password file I0507 06:33:49.882422 15263 server_base.cc:608] Dumped server information to /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_3.1557210500371863-231/raft_consensus-itest-cluster/ts-2/data/info.pb I0507 06:33:49.888787 231 external_mini_cluster.cc:995] Started /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver as pid 15263 I0507 06:33:49.893144 15386 heartbeater.cc:345] Connected to a master server at 127.0.57.254:43845 I0507 06:33:49.893395 15386 heartbeater.cc:426] Registering TS with master... I0507 06:33:49.893996 15386 heartbeater.cc:475] Master 127.0.57.254:43845 requested a full tablet report, sending... I0507 06:33:49.895635 14734 ts_manager.cc:159] Re-registered known tserver with Master: 8c9db460feac4f3ea7d87157753cc05f (127.0.57.195:38149) I0507 06:33:49.896129 14734 catalog_manager.cc:3390] Sending DeleteTablet(TABLET_DATA_TOMBSTONED) for tablet 0712c795dc4648ffa156a17cf52d071c on 8c9db460feac4f3ea7d87157753cc05f (127.0.57.195:38149) (Replica has no consensus available (current committed config index is 39)) I0507 06:33:49.896953 231 external_mini_cluster.cc:1172] Attempting to check leaks for /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-master pid 14711 I0507 06:33:49.897017 14734 master_service.cc:273] Signed X509 certificate for tserver {username='slave'} at 127.0.57.195:55227 I0507 06:33:49.897846 14767 generic_service.cc:171] Checking for leaks (request via RPC) I0507 06:33:49.994663 231 external_mini_cluster.cc:1131] Killing /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-master with pid 14711 I0507 06:33:50.013764 231 external_mini_cluster.cc:933] Running /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-master /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-master --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_3.1557210500371863-231/raft_consensus-itest-cluster/master-0/wal --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_3.1557210500371863-231/raft_consensus-itest-cluster/master-0/data --block_manager=log --webserver_interface=localhost --ipki_ca_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.0.57.254:43845 --webserver_interface=127.0.57.254 --webserver_port=35109 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_3.1557210500371863-231/raft_consensus-itest-cluster/master-0/logs --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_3.1557210500371863-231/raft_consensus-itest-cluster/master-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=true --master_add_server_when_underreplicated=false --rpc_reuseport=true --master_add_server_when_underreplicated=true with env {} W0507 06:33:50.014042 15391 negotiation.cc:313] Failed RPC negotiation. Trace: 0507 06:33:49.896777 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.0.0.1:51704 0507 06:33:49.899838 (+ 3061us) server_negotiation.cc:184] Beginning negotiation 0507 06:33:49.899844 (+ 6us) server_negotiation.cc:373] Waiting for connection header 0507 06:33:49.899872 (+ 28us) server_negotiation.cc:381] Connection header received 0507 06:33:49.899940 (+ 68us) server_negotiation.cc:337] Received NEGOTIATE NegotiatePB request 0507 06:33:49.899944 (+ 4us) server_negotiation.cc:420] Received NEGOTIATE request from client 0507 06:33:49.900115 (+ 171us) server_negotiation.cc:349] Sending NEGOTIATE NegotiatePB response 0507 06:33:49.900210 (+ 95us) server_negotiation.cc:205] Negotiated authn=CERTIFICATE 0507 06:33:49.994535 (+ 94325us) server_negotiation.cc:337] Received TLS_HANDSHAKE NegotiatePB request 0507 06:33:49.995154 (+ 619us) server_negotiation.cc:349] Sending TLS_HANDSHAKE NegotiatePB response 0507 06:33:50.012882 (+ 17728us) negotiation.cc:304] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51704: BlockingRecv error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107) Metrics: {"server-negotiator.queue_time_us":2899,"thread_start_us":2751,"threads_started":1} W0507 06:33:50.096055 15023 heartbeater.cc:587] Failed to heartbeat to 127.0.57.254:43845 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.57.254:43845: connect: Connection refused (error 111) W0507 06:33:50.104725 14900 heartbeater.cc:587] Failed to heartbeat to 127.0.57.254:43845 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.57.254:43845: connect: Connection refused (error 111) WARNING: Logging before InitGoogleLogging() is written to STDERR W0507 06:33:50.242452 15393 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0507 06:33:50.242712 15393 flags.cc:404] Enabled unsafe flag: --never_fsync=true W0507 06:33:50.245301 15393 flags.cc:404] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true W0507 06:33:50.245374 15393 flags.cc:404] Enabled experimental flag: --ipki_ca_key_size=1024 W0507 06:33:50.245437 15393 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W0507 06:33:50.245482 15393 flags.cc:404] Enabled experimental flag: --tsk_num_rsa_bits=512 W0507 06:33:50.245522 15393 flags.cc:404] Enabled experimental flag: --rpc_reuseport=true I0507 06:33:50.250843 15393 master_main.cc:73] Master server non-default flags: --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_3.1557210500371863-231/raft_consensus-itest-cluster/master-0/data --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_3.1557210500371863-231/raft_consensus-itest-cluster/master-0/wal --ipki_ca_key_size=1024 --ipki_server_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.0.57.254:43845 --rpc_reuseport=true --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_3.1557210500371863-231/raft_consensus-itest-cluster/master-0/data/info.pb --webserver_interface=127.0.57.254 --webserver_port=35109 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_3.1557210500371863-231/raft_consensus-itest-cluster/master-0/logs --logbuflevel=-1 --logtostderr=true Master server version: kudu 1.10.0-cdh6.x-SNAPSHOT revision 7d0ee26697926e87af9ef395008799577f6ab43f build type FASTDEBUG built by jenkins at 06 May 2019 22:44:21 PST on quasar-knmpfx-1.vpc.cloudera.com build id 322017 ASAN enabled I0507 06:33:50.258496 15393 system_ntp.cc:202] NTP initialized. Skew: 500ppm Current error: 189516us I0507 06:33:50.259954 15393 fs_manager.cc:263] Metadata directory not provided I0507 06:33:50.260041 15393 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0507 06:33:50.267304 15393 fs_manager.cc:399] Time spent opening directory manager: real 0.005s user 0.002s sys 0.002s I0507 06:33:50.268079 15393 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0507 06:33:50.268734 15393 file_cache.cc:466] Constructed file cache lbm with capacity 419430 I0507 06:33:50.272415 15393 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s I0507 06:33:50.272572 15393 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_3.1557210500371863-231/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_3.1557210500371863-231/raft_consensus-itest-cluster/master-0/wal uuid: "4b3cf1ac833f41f8ba8fb8623982ab03" format_stamp: "Formatted at 2019-05-07 06:33:28 on dist-test-slave-dist-test-slave-rqp6" I0507 06:33:50.273643 15393 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_3.1557210500371863-231/raft_consensus-itest-cluster/master-0/wal metadata directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_3.1557210500371863-231/raft_consensus-itest-cluster/master-0/wal 1 data directories: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_3.1557210500371863-231/raft_consensus-itest-cluster/master-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0507 06:33:50.299837 15393 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0507 06:33:50.316411 15393 rpc_server.cc:205] RPC server started. Bound to: 127.0.57.254:43845 I0507 06:33:50.317798 15393 webserver.cc:308] Webserver started at http://127.0.57.254:35109/ using document root and password file I0507 06:33:50.318763 15393 server_base.cc:608] Dumped server information to /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_3.1557210500371863-231/raft_consensus-itest-cluster/master-0/data/info.pb I0507 06:33:50.322909 231 external_mini_cluster.cc:995] Started /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-master as pid 15393 I0507 06:33:50.330066 15455 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 4b3cf1ac833f41f8ba8fb8623982ab03: Bootstrap starting. I0507 06:33:50.347859 15455 log.cc:525] T 00000000000000000000000000000000 P 4b3cf1ac833f41f8ba8fb8623982ab03: Log is configured to *not* fsync() on all Append() calls I0507 06:33:50.360039 15455 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 4b3cf1ac833f41f8ba8fb8623982ab03: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=4 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates I0507 06:33:50.360664 15455 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 4b3cf1ac833f41f8ba8fb8623982ab03: Bootstrap complete. I0507 06:33:50.368324 15455 raft_consensus.cc:340] T 00000000000000000000000000000000 P 4b3cf1ac833f41f8ba8fb8623982ab03 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b3cf1ac833f41f8ba8fb8623982ab03" member_type: VOTER } I0507 06:33:50.369293 15455 raft_consensus.cc:705] T 00000000000000000000000000000000 P 4b3cf1ac833f41f8ba8fb8623982ab03 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b3cf1ac833f41f8ba8fb8623982ab03, State: Initialized, Role: FOLLOWER I0507 06:33:50.369966 15455 consensus_queue.cc:229] T 00000000000000000000000000000000 P 4b3cf1ac833f41f8ba8fb8623982ab03 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b3cf1ac833f41f8ba8fb8623982ab03" member_type: VOTER } I0507 06:33:50.370149 15455 raft_consensus.cc:380] T 00000000000000000000000000000000 P 4b3cf1ac833f41f8ba8fb8623982ab03 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0507 06:33:50.370260 15455 raft_consensus.cc:472] T 00000000000000000000000000000000 P 4b3cf1ac833f41f8ba8fb8623982ab03 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0507 06:33:50.370394 15455 raft_consensus.cc:2886] T 00000000000000000000000000000000 P 4b3cf1ac833f41f8ba8fb8623982ab03 [term 1 FOLLOWER]: Advancing to term 2 I0507 06:33:50.375123 15455 raft_consensus.cc:494] T 00000000000000000000000000000000 P 4b3cf1ac833f41f8ba8fb8623982ab03 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b3cf1ac833f41f8ba8fb8623982ab03" member_type: VOTER } I0507 06:33:50.375710 15455 leader_election.cc:310] T 00000000000000000000000000000000 P 4b3cf1ac833f41f8ba8fb8623982ab03 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4b3cf1ac833f41f8ba8fb8623982ab03; no voters: I0507 06:33:50.376861 15459 raft_consensus.cc:2640] T 00000000000000000000000000000000 P 4b3cf1ac833f41f8ba8fb8623982ab03 [term 2 FOLLOWER]: Leader election won for term 2 I0507 06:33:50.377259 15455 leader_election.cc:296] T 00000000000000000000000000000000 P 4b3cf1ac833f41f8ba8fb8623982ab03 [CANDIDATE]: Term 2 election: Requested vote from peers I0507 06:33:50.377699 15459 raft_consensus.cc:667] T 00000000000000000000000000000000 P 4b3cf1ac833f41f8ba8fb8623982ab03 [term 2 LEADER]: Becoming Leader. State: Replica: 4b3cf1ac833f41f8ba8fb8623982ab03, State: Running, Role: LEADER I0507 06:33:50.378249 15459 consensus_queue.cc:206] T 00000000000000000000000000000000 P 4b3cf1ac833f41f8ba8fb8623982ab03 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b3cf1ac833f41f8ba8fb8623982ab03" member_type: VOTER } I0507 06:33:50.381114 15455 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4b3cf1ac833f41f8ba8fb8623982ab03 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 2 leader_uuid: "4b3cf1ac833f41f8ba8fb8623982ab03" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b3cf1ac833f41f8ba8fb8623982ab03" member_type: VOTER } } I0507 06:33:50.381477 15461 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4b3cf1ac833f41f8ba8fb8623982ab03 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "4b3cf1ac833f41f8ba8fb8623982ab03" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b3cf1ac833f41f8ba8fb8623982ab03" member_type: VOTER } } I0507 06:33:50.381693 15461 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4b3cf1ac833f41f8ba8fb8623982ab03 [sys.catalog]: This master's current role is: LEADER I0507 06:33:50.381429 15460 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4b3cf1ac833f41f8ba8fb8623982ab03 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4b3cf1ac833f41f8ba8fb8623982ab03. Latest consensus state: current_term: 2 leader_uuid: "4b3cf1ac833f41f8ba8fb8623982ab03" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b3cf1ac833f41f8ba8fb8623982ab03" member_type: VOTER } } I0507 06:33:50.381758 15460 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4b3cf1ac833f41f8ba8fb8623982ab03 [sys.catalog]: This master's current role is: LEADER I0507 06:33:50.382128 15455 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4b3cf1ac833f41f8ba8fb8623982ab03 [sys.catalog]: This master's current role is: LEADER I0507 06:33:50.383957 15455 sys_catalog.cc:424] T 00000000000000000000000000000000 P 4b3cf1ac833f41f8ba8fb8623982ab03 [sys.catalog]: configured and running, proceeding with master startup. I0507 06:33:50.384379 15464 catalog_manager.cc:1055] Loading table and tablet metadata into memory... I0507 06:33:50.395290 15464 catalog_manager.cc:356] Loaded metadata for table TestTable [id=7222e5f199b649a0876b51c5387cb8a9] I0507 06:33:50.397001 15464 catalog_manager.cc:410] Loaded metadata for tablet 0712c795dc4648ffa156a17cf52d071c (table TestTable [id=7222e5f199b649a0876b51c5387cb8a9]) I0507 06:33:50.397164 15464 catalog_manager.cc:1066] Initializing Kudu internal certificate authority... I0507 06:33:50.399967 15464 catalog_manager.cc:1075] Loading token signing keys... I0507 06:33:50.401125 15464 catalog_manager.cc:4303] T 00000000000000000000000000000000 P 4b3cf1ac833f41f8ba8fb8623982ab03: Loaded TSK: 0 I0507 06:33:50.899839 15386 heartbeater.cc:467] Master 127.0.57.254:43845 was elected leader, sending a full tablet report... I0507 06:33:50.907915 15422 master_service.cc:229] Got heartbeat from unknown tserver (permanent_uuid: "8c9db460feac4f3ea7d87157753cc05f" instance_seqno: 1557210829870813) as {username='slave'} at 127.0.57.195:58311; Asking this server to re-register. I0507 06:33:50.908885 15386 heartbeater.cc:426] Registering TS with master... I0507 06:33:50.909119 15386 heartbeater.cc:475] Master 127.0.57.254:43845 requested a full tablet report, sending... I0507 06:33:50.910465 15422 ts_manager.cc:159] Registered new tserver with Master: 8c9db460feac4f3ea7d87157753cc05f (127.0.57.195:38149) I0507 06:33:50.912331 15422 catalog_manager.cc:3390] Sending DeleteTablet(TABLET_DATA_TOMBSTONED) for tablet 0712c795dc4648ffa156a17cf52d071c on 8c9db460feac4f3ea7d87157753cc05f (127.0.57.195:38149) (Replica has no consensus available (current committed config index is 39)) I0507 06:33:50.920668 15322 tablet_service.cc:1057] Processing DeleteTablet for tablet 0712c795dc4648ffa156a17cf52d071c with delete_type TABLET_DATA_TOMBSTONED (Replica has no consensus available (current committed config index is 39)) from {username='slave'} at 127.0.0.1:51708 I0507 06:33:50.921584 15472 tablet_replica.cc:264] T 0712c795dc4648ffa156a17cf52d071c P 8c9db460feac4f3ea7d87157753cc05f: stopping tablet replica I0507 06:33:50.921979 15472 raft_consensus.cc:2143] T 0712c795dc4648ffa156a17cf52d071c P 8c9db460feac4f3ea7d87157753cc05f [term 1 FOLLOWER]: Raft consensus shutting down. I0507 06:33:50.922297 15472 raft_consensus.cc:2170] T 0712c795dc4648ffa156a17cf52d071c P 8c9db460feac4f3ea7d87157753cc05f [term 1 FOLLOWER]: Raft consensus is shut down! I0507 06:33:50.923552 15472 ts_tablet_manager.cc:1377] T 0712c795dc4648ffa156a17cf52d071c P 8c9db460feac4f3ea7d87157753cc05f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED I0507 06:33:50.928926 15472 ts_tablet_manager.cc:1390] T 0712c795dc4648ffa156a17cf52d071c P 8c9db460feac4f3ea7d87157753cc05f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.1 I0507 06:33:50.929021 15472 log.cc:990] T 0712c795dc4648ffa156a17cf52d071c P 8c9db460feac4f3ea7d87157753cc05f: Deleting WAL directory at /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_3.1557210500371863-231/raft_consensus-itest-cluster/ts-2/wal/wals/0712c795dc4648ffa156a17cf52d071c I0507 06:33:50.930464 15405 catalog_manager.cc:3368] TS 8c9db460feac4f3ea7d87157753cc05f (127.0.57.195:38149): tablet 0712c795dc4648ffa156a17cf52d071c (table TestTable [id=7222e5f199b649a0876b51c5387cb8a9]) successfully deleted I0507 06:33:51.103519 15023 heartbeater.cc:345] Connected to a master server at 127.0.57.254:43845 I0507 06:33:51.104519 15422 master_service.cc:229] Got heartbeat from unknown tserver (permanent_uuid: "9d7b3cf9bef345e38290f5f6ec1b3a12" instance_seqno: 1557210809572025) as {username='slave'} at 127.0.57.194:46735; Asking this server to re-register. I0507 06:33:51.105396 15023 heartbeater.cc:426] Registering TS with master... I0507 06:33:51.105595 15023 heartbeater.cc:475] Master 127.0.57.254:43845 requested a full tablet report, sending... I0507 06:33:51.106827 15422 ts_manager.cc:159] Registered new tserver with Master: 9d7b3cf9bef345e38290f5f6ec1b3a12 (127.0.57.194:38001) I0507 06:33:51.110961 14900 heartbeater.cc:345] Connected to a master server at 127.0.57.254:43845 I0507 06:33:51.111801 15422 master_service.cc:229] Got heartbeat from unknown tserver (permanent_uuid: "69ac1841085d4196ac900d65a0238f79" instance_seqno: 1557210809285594) as {username='slave'} at 127.0.57.193:38091; Asking this server to re-register. I0507 06:33:51.112326 14900 heartbeater.cc:426] Registering TS with master... I0507 06:33:51.112481 14900 heartbeater.cc:475] Master 127.0.57.254:43845 requested a full tablet report, sending... I0507 06:33:51.113466 15422 ts_manager.cc:159] Registered new tserver with Master: 69ac1841085d4196ac900d65a0238f79 (127.0.57.193:43169) I0507 06:33:51.114300 15422 catalog_manager.cc:3616] Sending ChangeConfig:ADD_PEER:NON_VOTER on tablet 0712c795dc4648ffa156a17cf52d071c (attempt 1) I0507 06:33:51.121868 14857 consensus_queue.cc:206] T 0712c795dc4648ffa156a17cf52d071c P 69ac1841085d4196ac900d65a0238f79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 39, Committed index: 39, Last appended: 2.39, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 40 OBSOLETE_local: false peers { permanent_uuid: "9d7b3cf9bef345e38290f5f6ec1b3a12" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 38001 } } peers { permanent_uuid: "69ac1841085d4196ac900d65a0238f79" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 43169 } } peers { permanent_uuid: "8c9db460feac4f3ea7d87157753cc05f" member_type: NON_VOTER last_known_addr { host: "127.0.57.195" port: 38149 } attrs { promote: true } } I0507 06:33:51.124950 14979 raft_consensus.cc:1181] T 0712c795dc4648ffa156a17cf52d071c P 9d7b3cf9bef345e38290f5f6ec1b3a12 [term 2 FOLLOWER]: Refusing update from remote peer 69ac1841085d4196ac900d65a0238f79: Log matching property violated. Preceding OpId in replica: term: 2 index: 39. Preceding OpId from leader: term: 2 index: 40. (index mismatch) I0507 06:33:51.125581 15467 consensus_queue.cc:984] T 0712c795dc4648ffa156a17cf52d071c P 69ac1841085d4196ac900d65a0238f79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d7b3cf9bef345e38290f5f6ec1b3a12" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 38001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 40, Last known committed idx: 39, Time since last communication: 0.000s I0507 06:33:51.129513 15477 raft_consensus.cc:2788] T 0712c795dc4648ffa156a17cf52d071c P 69ac1841085d4196ac900d65a0238f79 [term 2 LEADER]: Committing config change with OpId 2.40: config changed from index 39 to 40, NON_VOTER 8c9db460feac4f3ea7d87157753cc05f (127.0.57.195) added. New config: { opid_index: 40 OBSOLETE_local: false peers { permanent_uuid: "9d7b3cf9bef345e38290f5f6ec1b3a12" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 38001 } } peers { permanent_uuid: "69ac1841085d4196ac900d65a0238f79" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 43169 } } peers { permanent_uuid: "8c9db460feac4f3ea7d87157753cc05f" member_type: NON_VOTER last_known_addr { host: "127.0.57.195" port: 38149 } attrs { promote: true } } } W0507 06:33:51.130534 14790 consensus_peers.cc:457] T 0712c795dc4648ffa156a17cf52d071c P 69ac1841085d4196ac900d65a0238f79 -> Peer 8c9db460feac4f3ea7d87157753cc05f (127.0.57.195:38149): Couldn't send request to peer 8c9db460feac4f3ea7d87157753cc05f. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry. I0507 06:33:51.130543 14979 raft_consensus.cc:2788] T 0712c795dc4648ffa156a17cf52d071c P 9d7b3cf9bef345e38290f5f6ec1b3a12 [term 2 FOLLOWER]: Committing config change with OpId 2.40: config changed from index 39 to 40, NON_VOTER 8c9db460feac4f3ea7d87157753cc05f (127.0.57.195) added. New config: { opid_index: 40 OBSOLETE_local: false peers { permanent_uuid: "9d7b3cf9bef345e38290f5f6ec1b3a12" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 38001 } } peers { permanent_uuid: "69ac1841085d4196ac900d65a0238f79" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 43169 } } peers { permanent_uuid: "8c9db460feac4f3ea7d87157753cc05f" member_type: NON_VOTER last_known_addr { host: "127.0.57.195" port: 38149 } attrs { promote: true } } } I0507 06:33:51.133138 15405 catalog_manager.cc:3531] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 0712c795dc4648ffa156a17cf52d071c with cas_config_opid_index 39: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1) I0507 06:33:51.135025 15422 catalog_manager.cc:3988] T 0712c795dc4648ffa156a17cf52d071c P 69ac1841085d4196ac900d65a0238f79 reported cstate change: config changed from index 39 to 40, NON_VOTER 8c9db460feac4f3ea7d87157753cc05f (127.0.57.195) added. New cstate: current_term: 2 leader_uuid: "69ac1841085d4196ac900d65a0238f79" committed_config { opid_index: 40 OBSOLETE_local: false peers { permanent_uuid: "9d7b3cf9bef345e38290f5f6ec1b3a12" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 38001 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "69ac1841085d4196ac900d65a0238f79" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 43169 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8c9db460feac4f3ea7d87157753cc05f" member_type: NON_VOTER last_known_addr { host: "127.0.57.195" port: 38149 } attrs { promote: true } health_report { overall_health: UNKNOWN } } } I0507 06:33:51.523752 15484 ts_tablet_manager.cc:676] T 0712c795dc4648ffa156a17cf52d071c P 8c9db460feac4f3ea7d87157753cc05f: Initiating tablet copy from peer 69ac1841085d4196ac900d65a0238f79 (127.0.57.193:43169) I0507 06:33:51.524204 15484 tablet_copy_client.cc:240] T 0712c795dc4648ffa156a17cf52d071c P 8c9db460feac4f3ea7d87157753cc05f: tablet copy: Beginning tablet copy session from remote peer at address 127.0.57.193:43169 I0507 06:33:51.535450 14877 tablet_copy_service.cc:135] P 69ac1841085d4196ac900d65a0238f79: Received BeginTabletCopySession request for tablet 0712c795dc4648ffa156a17cf52d071c from peer 8c9db460feac4f3ea7d87157753cc05f ({username='slave'} at 127.0.57.195:49479) I0507 06:33:51.535635 14877 tablet_copy_service.cc:156] P 69ac1841085d4196ac900d65a0238f79: Beginning new tablet copy session on tablet 0712c795dc4648ffa156a17cf52d071c from peer 8c9db460feac4f3ea7d87157753cc05f at {username='slave'} at 127.0.57.195:49479: session id = 8c9db460feac4f3ea7d87157753cc05f-0712c795dc4648ffa156a17cf52d071c I0507 06:33:51.538404 14877 tablet_copy_source_session.cc:202] T 0712c795dc4648ffa156a17cf52d071c P 69ac1841085d4196ac900d65a0238f79: Tablet Copy: opened 5 blocks and 2 log segments I0507 06:33:51.540805 15484 ts_tablet_manager.cc:1377] T 0712c795dc4648ffa156a17cf52d071c P 8c9db460feac4f3ea7d87157753cc05f: Deleting tablet data with delete state TABLET_DATA_COPYING I0507 06:33:51.545040 15484 ts_tablet_manager.cc:1390] T 0712c795dc4648ffa156a17cf52d071c P 8c9db460feac4f3ea7d87157753cc05f: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.1 I0507 06:33:51.545423 15484 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet 0712c795dc4648ffa156a17cf52d071c. 1 dirs total, 0 dirs full, 0 dirs failed I0507 06:33:51.547777 15484 tablet_copy_client.cc:575] T 0712c795dc4648ffa156a17cf52d071c P 8c9db460feac4f3ea7d87157753cc05f: tablet copy: Starting download of 5 data blocks... I0507 06:33:51.559060 15484 tablet_copy_client.cc:538] T 0712c795dc4648ffa156a17cf52d071c P 8c9db460feac4f3ea7d87157753cc05f: tablet copy: Starting download of 2 WAL segments... I0507 06:33:51.570113 15484 tablet_copy_client.cc:414] T 0712c795dc4648ffa156a17cf52d071c P 8c9db460feac4f3ea7d87157753cc05f: tablet copy: Tablet Copy complete. Replacing tablet superblock. I0507 06:33:51.572962 15484 tablet_bootstrap.cc:439] T 0712c795dc4648ffa156a17cf52d071c P 8c9db460feac4f3ea7d87157753cc05f: Bootstrap starting. I0507 06:33:51.590533 15484 tablet_bootstrap.cc:439] T 0712c795dc4648ffa156a17cf52d071c P 8c9db460feac4f3ea7d87157753cc05f: Bootstrap replayed 1/2 log segments. Stats: ops{read=7 overwritten=0 applied=6 ignored=6} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=3. Pending: 1 replicates I0507 06:33:51.596839 15484 tablet_bootstrap.cc:439] T 0712c795dc4648ffa156a17cf52d071c P 8c9db460feac4f3ea7d87157753cc05f: Bootstrap replayed 2/2 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=6} inserts{seen=2 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=3. Pending: 0 replicates I0507 06:33:51.597237 15484 tablet_bootstrap.cc:439] T 0712c795dc4648ffa156a17cf52d071c P 8c9db460feac4f3ea7d87157753cc05f: Bootstrap complete. I0507 06:33:51.597646 15484 ts_tablet_manager.cc:1053] T 0712c795dc4648ffa156a17cf52d071c P 8c9db460feac4f3ea7d87157753cc05f: Time spent bootstrapping tablet: real 0.025s user 0.023s sys 0.000s I0507 06:33:51.598361 15484 raft_consensus.cc:340] T 0712c795dc4648ffa156a17cf52d071c P 8c9db460feac4f3ea7d87157753cc05f [term 2 LEARNER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: 40 OBSOLETE_local: false peers { permanent_uuid: "9d7b3cf9bef345e38290f5f6ec1b3a12" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 38001 } } peers { permanent_uuid: "69ac1841085d4196ac900d65a0238f79" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 43169 } } peers { permanent_uuid: "8c9db460feac4f3ea7d87157753cc05f" member_type: NON_VOTER last_known_addr { host: "127.0.57.195" port: 38149 } attrs { promote: true } } I0507 06:33:51.598556 15484 raft_consensus.cc:705] T 0712c795dc4648ffa156a17cf52d071c P 8c9db460feac4f3ea7d87157753cc05f [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 8c9db460feac4f3ea7d87157753cc05f, State: Initialized, Role: LEARNER I0507 06:33:51.598680 15484 consensus_queue.cc:229] T 0712c795dc4648ffa156a17cf52d071c P 8c9db460feac4f3ea7d87157753cc05f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 40, Last appended: 2.40, Last appended by leader: 40, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 40 OBSOLETE_local: false peers { permanent_uuid: "9d7b3cf9bef345e38290f5f6ec1b3a12" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 38001 } } peers { permanent_uuid: "69ac1841085d4196ac900d65a0238f79" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 43169 } } peers { permanent_uuid: "8c9db460feac4f3ea7d87157753cc05f" member_type: NON_VOTER last_known_addr { host: "127.0.57.195" port: 38149 } attrs { promote: true } } I0507 06:33:51.599465 15484 ts_tablet_manager.cc:1081] T 0712c795dc4648ffa156a17cf52d071c P 8c9db460feac4f3ea7d87157753cc05f: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s I0507 06:33:51.600620 14877 tablet_copy_service.cc:325] P 69ac1841085d4196ac900d65a0238f79: Request end of tablet copy session 8c9db460feac4f3ea7d87157753cc05f-0712c795dc4648ffa156a17cf52d071c received from {username='slave'} at 127.0.57.195:49479 I0507 06:33:51.600767 14877 tablet_copy_service.cc:413] P 69ac1841085d4196ac900d65a0238f79: ending tablet copy session 8c9db460feac4f3ea7d87157753cc05f-0712c795dc4648ffa156a17cf52d071c on tablet 0712c795dc4648ffa156a17cf52d071c with peer 8c9db460feac4f3ea7d87157753cc05f I0507 06:33:51.690990 15387 maintenance_manager.cc:306] P 8c9db460feac4f3ea7d87157753cc05f: Scheduling UndoDeltaBlockGCOp(0712c795dc4648ffa156a17cf52d071c): 516 bytes on disk I0507 06:33:51.692801 15278 maintenance_manager.cc:501] P 8c9db460feac4f3ea7d87157753cc05f: UndoDeltaBlockGCOp(0712c795dc4648ffa156a17cf52d071c) complete. Timing: real 0.001s user 0.001s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":98,"lbm_reads_lt_1ms":4} I0507 06:33:52.062749 15342 raft_consensus.cc:1126] T 0712c795dc4648ffa156a17cf52d071c P 8c9db460feac4f3ea7d87157753cc05f [term 2 LEARNER]: Deduplicated request from leader. Original: 2.39->[2.40-2.40] Dedup: 2.40->[] I0507 06:33:52.450000 15488 raft_consensus.cc:974] T 0712c795dc4648ffa156a17cf52d071c P 69ac1841085d4196ac900d65a0238f79: attempting to promote NON_VOTER 8c9db460feac4f3ea7d87157753cc05f to VOTER I0507 06:33:52.450716 15488 consensus_queue.cc:206] T 0712c795dc4648ffa156a17cf52d071c P 69ac1841085d4196ac900d65a0238f79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 40, Committed index: 40, Last appended: 2.40, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 41 OBSOLETE_local: false peers { permanent_uuid: "9d7b3cf9bef345e38290f5f6ec1b3a12" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 38001 } } peers { permanent_uuid: "69ac1841085d4196ac900d65a0238f79" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 43169 } } peers { permanent_uuid: "8c9db460feac4f3ea7d87157753cc05f" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 38149 } attrs { promote: false } } I0507 06:33:52.453045 15342 raft_consensus.cc:1181] T 0712c795dc4648ffa156a17cf52d071c P 8c9db460feac4f3ea7d87157753cc05f [term 2 LEARNER]: Refusing update from remote peer 69ac1841085d4196ac900d65a0238f79: Log matching property violated. Preceding OpId in replica: term: 2 index: 40. Preceding OpId from leader: term: 2 index: 41. (index mismatch) I0507 06:33:52.453131 14979 raft_consensus.cc:1181] T 0712c795dc4648ffa156a17cf52d071c P 9d7b3cf9bef345e38290f5f6ec1b3a12 [term 2 FOLLOWER]: Refusing update from remote peer 69ac1841085d4196ac900d65a0238f79: Log matching property violated. Preceding OpId in replica: term: 2 index: 40. Preceding OpId from leader: term: 2 index: 41. (index mismatch) I0507 06:33:52.453743 15489 consensus_queue.cc:984] T 0712c795dc4648ffa156a17cf52d071c P 69ac1841085d4196ac900d65a0238f79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d7b3cf9bef345e38290f5f6ec1b3a12" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 38001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 41, Last known committed idx: 40, Time since last communication: 0.000s I0507 06:33:52.454051 15488 consensus_queue.cc:984] T 0712c795dc4648ffa156a17cf52d071c P 69ac1841085d4196ac900d65a0238f79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c9db460feac4f3ea7d87157753cc05f" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 38149 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 41, Last known committed idx: 40, Time since last communication: 0.000s I0507 06:33:52.457780 15489 raft_consensus.cc:2788] T 0712c795dc4648ffa156a17cf52d071c P 69ac1841085d4196ac900d65a0238f79 [term 2 LEADER]: Committing config change with OpId 2.41: config changed from index 40 to 41, 8c9db460feac4f3ea7d87157753cc05f (127.0.57.195) changed from NON_VOTER to VOTER. New config: { opid_index: 41 OBSOLETE_local: false peers { permanent_uuid: "9d7b3cf9bef345e38290f5f6ec1b3a12" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 38001 } } peers { permanent_uuid: "69ac1841085d4196ac900d65a0238f79" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 43169 } } peers { permanent_uuid: "8c9db460feac4f3ea7d87157753cc05f" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 38149 } attrs { promote: false } } } I0507 06:33:52.458504 15342 raft_consensus.cc:2788] T 0712c795dc4648ffa156a17cf52d071c P 8c9db460feac4f3ea7d87157753cc05f [term 2 FOLLOWER]: Committing config change with OpId 2.41: config changed from index 40 to 41, 8c9db460feac4f3ea7d87157753cc05f (127.0.57.195) changed from NON_VOTER to VOTER. New config: { opid_index: 41 OBSOLETE_local: false peers { permanent_uuid: "9d7b3cf9bef345e38290f5f6ec1b3a12" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 38001 } } peers { permanent_uuid: "69ac1841085d4196ac900d65a0238f79" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 43169 } } peers { permanent_uuid: "8c9db460feac4f3ea7d87157753cc05f" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 38149 } attrs { promote: false } } } I0507 06:33:52.462690 15421 catalog_manager.cc:3988] T 0712c795dc4648ffa156a17cf52d071c P 69ac1841085d4196ac900d65a0238f79 reported cstate change: config changed from index 40 to 41, 8c9db460feac4f3ea7d87157753cc05f (127.0.57.195) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "69ac1841085d4196ac900d65a0238f79" committed_config { opid_index: 41 OBSOLETE_local: false peers { permanent_uuid: "9d7b3cf9bef345e38290f5f6ec1b3a12" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 38001 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "69ac1841085d4196ac900d65a0238f79" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 43169 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8c9db460feac4f3ea7d87157753cc05f" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 38149 } attrs { promote: false } health_report { overall_health: HEALTHY } } } I0507 06:33:52.463407 14979 raft_consensus.cc:2788] T 0712c795dc4648ffa156a17cf52d071c P 9d7b3cf9bef345e38290f5f6ec1b3a12 [term 2 FOLLOWER]: Committing config change with OpId 2.41: config changed from index 40 to 41, 8c9db460feac4f3ea7d87157753cc05f (127.0.57.195) changed from NON_VOTER to VOTER. New config: { opid_index: 41 OBSOLETE_local: false peers { permanent_uuid: "9d7b3cf9bef345e38290f5f6ec1b3a12" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 38001 } } peers { permanent_uuid: "69ac1841085d4196ac900d65a0238f79" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 43169 } } peers { permanent_uuid: "8c9db460feac4f3ea7d87157753cc05f" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 38149 } attrs { promote: false } } } Master Summary UUID | Address | Status ----------------------------------+--------------------+--------- 4b3cf1ac833f41f8ba8fb8623982ab03 | 127.0.57.254:43845 | HEALTHY Flag | Value | Tags | Master ------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+------------------------- ipki_ca_key_size | 1024 | experimental | all 1 server(s) checked ipki_server_key_size | 1024 | experimental | all 1 server(s) checked master_add_server_when_underreplicated | true | hidden | all 1 server(s) checked never_fsync | true | unsafe,advanced | all 1 server(s) checked raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked rpc_reuseport | true | experimental | all 1 server(s) checked rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked server_dump_info_format | pb | hidden | all 1 server(s) checked server_dump_info_path | /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_3.1557210500371863-231/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked Tablet Server Summary UUID | Address | Status | Location ----------------------------------+--------------------+---------+---------- 69ac1841085d4196ac900d65a0238f79 | 127.0.57.193:43169 | HEALTHY | 8c9db460feac4f3ea7d87157753cc05f | 127.0.57.195:38149 | HEALTHY | 9d7b3cf9bef345e38290f5f6ec1b3a12 | 127.0.57.194:38001 | HEALTHY | Tablet Server Location Summary Location | Count ----------+--------- | 3 Flag | Value | Tags | Tablet Server ------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------------------------------+------------------------- ipki_server_key_size | 1024 | experimental | all 3 server(s) checked local_ip_for_outbound_sockets | 127.0.57.193 | experimental | 127.0.57.193:43169 local_ip_for_outbound_sockets | 127.0.57.194 | experimental | 127.0.57.194:38001 local_ip_for_outbound_sockets | 127.0.57.195 | experimental | 127.0.57.195:38149 log_compression_codec | none | experimental | all 3 server(s) checked log_max_segments_to_retain | 3 | experimental,advanced,runtime | all 3 server(s) checked log_target_replay_size_mb | 1 | experimental | all 3 server(s) checked maintenance_manager_polling_interval_ms | 100 | hidden | all 3 server(s) checked max_cell_size_bytes | 1000000 | unsafe | all 3 server(s) checked never_fsync | true | unsafe,advanced | all 3 server(s) checked raft_prepare_replacement_before_eviction | true | experimental,advanced | all 3 server(s) checked rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked server_dump_info_format | pb | hidden | all 3 server(s) checked server_dump_info_path | /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_3.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.0.57.193:43169 server_dump_info_path | /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_3.1557210500371863-231/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.0.57.194:38001 server_dump_info_path | /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_3.1557210500371863-231/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.0.57.195:38149 Version Summary Version | Servers ------------------------+------------------------- 1.10.0-cdh6.x-SNAPSHOT | all 4 server(s) checked Tablet Summary Summary by table Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable -----------+----+---------+---------------+---------+------------+------------------+------------- TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0 Tablet Replica Count Summary Statistic | Replica Count ----------------+--------------- Minimum | 1 First Quartile | 1 Median | 1 Third Quartile | 1 Maximum | 1 Total Count Summary | Total Count ----------------+------------- Masters | 1 Tablet Servers | 3 Tables | 1 Tablets | 1 Replicas | 3 ================== Warnings: ================== Some masters have unsafe, experimental, or hidden flags set Some tablet servers have unsafe, experimental, or hidden flags set OK I0507 06:33:52.582046 231 log_verifier.cc:121] Checking tablet 0712c795dc4648ffa156a17cf52d071c I0507 06:33:52.598392 231 log_verifier.cc:172] Verified matching terms for 14 ops in tablet 0712c795dc4648ffa156a17cf52d071c I0507 06:33:52.599158 231 external_mini_cluster.cc:1172] Attempting to check leaks for /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver pid 14771 I0507 06:33:52.600513 14897 generic_service.cc:171] Checking for leaks (request via RPC) I0507 06:33:52.692028 231 external_mini_cluster.cc:1131] Killing /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver with pid 14771 I0507 06:33:52.710142 231 external_mini_cluster.cc:1172] Attempting to check leaks for /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver pid 14904 I0507 06:33:52.711309 15020 generic_service.cc:171] Checking for leaks (request via RPC) I0507 06:33:52.800423 231 external_mini_cluster.cc:1131] Killing /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver with pid 14904 I0507 06:33:52.820039 231 external_mini_cluster.cc:1172] Attempting to check leaks for /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver pid 15263 I0507 06:33:52.825927 15383 generic_service.cc:171] Checking for leaks (request via RPC) I0507 06:33:52.881081 231 external_mini_cluster.cc:1131] Killing /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver with pid 15263 I0507 06:33:52.892141 231 external_mini_cluster.cc:1172] Attempting to check leaks for /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-master pid 15393 I0507 06:33:52.893218 15452 generic_service.cc:171] Checking for leaks (request via RPC) I0507 06:33:52.969780 231 external_mini_cluster.cc:1131] Killing /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-master with pid 15393 [ OK ] ReplicaBehindWalGcThresholdITest.ReplicaReplacement/3 (24332 ms) [ RUN ] ReplicaBehindWalGcThresholdITest.ReplicaReplacement/4 I0507 06:33:52.994452 231 test_util.cc:212] Using random seed: 1720352896 I0507 06:33:52.995939 231 ts_itest-base.cc:112] Starting cluster with: I0507 06:33:52.995975 231 ts_itest-base.cc:113] -------------- I0507 06:33:52.995992 231 ts_itest-base.cc:114] 3 tablet servers I0507 06:33:52.996009 231 ts_itest-base.cc:115] 3 replicas per TS I0507 06:33:52.996055 231 ts_itest-base.cc:116] -------------- I0507 06:33:52.997121 231 external_mini_cluster.cc:933] Running /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-master /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-master --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_4.1557210500371863-231/raft_consensus-itest-cluster/master-0/wal --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_4.1557210500371863-231/raft_consensus-itest-cluster/master-0/data --block_manager=log --webserver_interface=localhost --ipki_ca_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.0.57.254:34189 --webserver_interface=127.0.57.254 --webserver_port=0 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_4.1557210500371863-231/raft_consensus-itest-cluster/master-0/logs --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_4.1557210500371863-231/raft_consensus-itest-cluster/master-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=true --rpc_reuseport=true with env {} WARNING: Logging before InitGoogleLogging() is written to STDERR W0507 06:33:53.203835 15534 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0507 06:33:53.204056 15534 flags.cc:404] Enabled unsafe flag: --never_fsync=true W0507 06:33:53.206518 15534 flags.cc:404] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true W0507 06:33:53.206578 15534 flags.cc:404] Enabled experimental flag: --ipki_ca_key_size=1024 W0507 06:33:53.206614 15534 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W0507 06:33:53.206635 15534 flags.cc:404] Enabled experimental flag: --tsk_num_rsa_bits=512 W0507 06:33:53.206677 15534 flags.cc:404] Enabled experimental flag: --rpc_reuseport=true I0507 06:33:53.211069 15534 master_main.cc:73] Master server non-default flags: --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_4.1557210500371863-231/raft_consensus-itest-cluster/master-0/data --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_4.1557210500371863-231/raft_consensus-itest-cluster/master-0/wal --ipki_ca_key_size=1024 --ipki_server_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.0.57.254:34189 --rpc_reuseport=true --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_4.1557210500371863-231/raft_consensus-itest-cluster/master-0/data/info.pb --webserver_interface=127.0.57.254 --webserver_port=0 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_4.1557210500371863-231/raft_consensus-itest-cluster/master-0/logs --logbuflevel=-1 --logtostderr=true Master server version: kudu 1.10.0-cdh6.x-SNAPSHOT revision 7d0ee26697926e87af9ef395008799577f6ab43f build type FASTDEBUG built by jenkins at 06 May 2019 22:44:21 PST on quasar-knmpfx-1.vpc.cloudera.com build id 322017 ASAN enabled I0507 06:33:53.217569 15534 system_ntp.cc:202] NTP initialized. Skew: 500ppm Current error: 191016us I0507 06:33:53.218776 15534 fs_manager.cc:263] Metadata directory not provided I0507 06:33:53.218840 15534 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0507 06:33:53.219110 15534 server_base.cc:433] Could not load existing FS layout: Not found: could not find a healthy instance file I0507 06:33:53.219174 15534 server_base.cc:434] Attempting to create new FS layout instead I0507 06:33:53.221635 15534 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_4.1557210500371863-231/raft_consensus-itest-cluster/master-0/data/instance: uuid: "96510d60d2b748329ce1e8d69faaad41" format_stamp: "Formatted at 2019-05-07 06:33:53 on dist-test-slave-dist-test-slave-rqp6" I0507 06:33:53.222151 15534 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_4.1557210500371863-231/raft_consensus-itest-cluster/master-0/wal/instance: uuid: "96510d60d2b748329ce1e8d69faaad41" format_stamp: "Formatted at 2019-05-07 06:33:53 on dist-test-slave-dist-test-slave-rqp6" I0507 06:33:53.229859 15534 fs_manager.cc:503] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s I0507 06:33:53.230960 15534 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0507 06:33:53.231531 15534 file_cache.cc:466] Constructed file cache lbm with capacity 419430 I0507 06:33:53.235219 15534 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s I0507 06:33:53.235306 15534 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_4.1557210500371863-231/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_4.1557210500371863-231/raft_consensus-itest-cluster/master-0/wal uuid: "96510d60d2b748329ce1e8d69faaad41" format_stamp: "Formatted at 2019-05-07 06:33:53 on dist-test-slave-dist-test-slave-rqp6" I0507 06:33:53.235455 15534 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_4.1557210500371863-231/raft_consensus-itest-cluster/master-0/wal metadata directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_4.1557210500371863-231/raft_consensus-itest-cluster/master-0/wal 1 data directories: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_4.1557210500371863-231/raft_consensus-itest-cluster/master-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0507 06:33:53.260020 15534 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0507 06:33:53.274832 15534 rpc_server.cc:205] RPC server started. Bound to: 127.0.57.254:34189 I0507 06:33:53.275895 15534 webserver.cc:308] Webserver started at http://127.0.57.254:46335/ using document root and password file I0507 06:33:53.276607 15534 server_base.cc:608] Dumped server information to /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_4.1557210500371863-231/raft_consensus-itest-cluster/master-0/data/info.pb I0507 06:33:53.279867 15593 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0507 06:33:53.283529 231 external_mini_cluster.cc:995] Started /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-master as pid 15534 I0507 06:33:53.284337 231 external_mini_cluster.cc:933] Running /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_4.1557210500371863-231/raft_consensus-itest-cluster/ts-0/wal --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_4.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data --block_manager=log --rpc_bind_addresses=127.0.57.193:0 --local_ip_for_outbound_sockets=127.0.57.193 --webserver_interface=127.0.57.193 --webserver_port=0 --tserver_master_addrs=127.0.57.254:34189 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_4.1557210500371863-231/raft_consensus-itest-cluster/ts-0/logs --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_4.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=true --follower_unavailable_considered_failed_sec=5 --log_compression_codec=none --log_cache_size_limit_mb=1 --log_segment_size_mb=1 --log_async_preallocate_segments=false --log_min_segments_to_retain=1 --log_max_segments_to_retain=3 --maintenance_manager_polling_interval_ms=100 --log_target_replay_size_mb=1 --max_cell_size_bytes=1000000 with env {} I0507 06:33:53.290588 15593 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 96510d60d2b748329ce1e8d69faaad41: Bootstrap starting. I0507 06:33:53.293159 15593 tablet_bootstrap.cc:590] T 00000000000000000000000000000000 P 96510d60d2b748329ce1e8d69faaad41: No blocks or log segments found. Creating new log. I0507 06:33:53.294044 15593 log.cc:525] T 00000000000000000000000000000000 P 96510d60d2b748329ce1e8d69faaad41: Log is configured to *not* fsync() on all Append() calls I0507 06:33:53.295929 15593 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 96510d60d2b748329ce1e8d69faaad41: No bootstrap required, opened a new log I0507 06:33:53.310741 15593 raft_consensus.cc:340] T 00000000000000000000000000000000 P 96510d60d2b748329ce1e8d69faaad41 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96510d60d2b748329ce1e8d69faaad41" member_type: VOTER } I0507 06:33:53.311009 15593 raft_consensus.cc:366] T 00000000000000000000000000000000 P 96510d60d2b748329ce1e8d69faaad41 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0507 06:33:53.311070 15593 raft_consensus.cc:705] T 00000000000000000000000000000000 P 96510d60d2b748329ce1e8d69faaad41 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96510d60d2b748329ce1e8d69faaad41, State: Initialized, Role: FOLLOWER I0507 06:33:53.311617 15593 consensus_queue.cc:229] T 00000000000000000000000000000000 P 96510d60d2b748329ce1e8d69faaad41 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96510d60d2b748329ce1e8d69faaad41" member_type: VOTER } I0507 06:33:53.311848 15593 raft_consensus.cc:380] T 00000000000000000000000000000000 P 96510d60d2b748329ce1e8d69faaad41 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0507 06:33:53.311924 15593 raft_consensus.cc:472] T 00000000000000000000000000000000 P 96510d60d2b748329ce1e8d69faaad41 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0507 06:33:53.312022 15593 raft_consensus.cc:2886] T 00000000000000000000000000000000 P 96510d60d2b748329ce1e8d69faaad41 [term 0 FOLLOWER]: Advancing to term 1 I0507 06:33:53.315037 15593 raft_consensus.cc:494] T 00000000000000000000000000000000 P 96510d60d2b748329ce1e8d69faaad41 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96510d60d2b748329ce1e8d69faaad41" member_type: VOTER } I0507 06:33:53.315733 15593 leader_election.cc:310] T 00000000000000000000000000000000 P 96510d60d2b748329ce1e8d69faaad41 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 96510d60d2b748329ce1e8d69faaad41; no voters: I0507 06:33:53.316758 15593 leader_election.cc:296] T 00000000000000000000000000000000 P 96510d60d2b748329ce1e8d69faaad41 [CANDIDATE]: Term 1 election: Requested vote from peers I0507 06:33:53.316983 15596 raft_consensus.cc:2640] T 00000000000000000000000000000000 P 96510d60d2b748329ce1e8d69faaad41 [term 1 FOLLOWER]: Leader election won for term 1 I0507 06:33:53.321070 15596 raft_consensus.cc:667] T 00000000000000000000000000000000 P 96510d60d2b748329ce1e8d69faaad41 [term 1 LEADER]: Becoming Leader. State: Replica: 96510d60d2b748329ce1e8d69faaad41, State: Running, Role: LEADER I0507 06:33:53.321568 15596 consensus_queue.cc:206] T 00000000000000000000000000000000 P 96510d60d2b748329ce1e8d69faaad41 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96510d60d2b748329ce1e8d69faaad41" member_type: VOTER } I0507 06:33:53.325781 15593 sys_catalog.cc:337] T 00000000000000000000000000000000 P 96510d60d2b748329ce1e8d69faaad41 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 1 leader_uuid: "96510d60d2b748329ce1e8d69faaad41" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96510d60d2b748329ce1e8d69faaad41" member_type: VOTER } } I0507 06:33:53.326810 15593 sys_catalog.cc:340] T 00000000000000000000000000000000 P 96510d60d2b748329ce1e8d69faaad41 [sys.catalog]: This master's current role is: LEADER I0507 06:33:53.326709 15598 sys_catalog.cc:337] T 00000000000000000000000000000000 P 96510d60d2b748329ce1e8d69faaad41 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 96510d60d2b748329ce1e8d69faaad41. Latest consensus state: current_term: 1 leader_uuid: "96510d60d2b748329ce1e8d69faaad41" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96510d60d2b748329ce1e8d69faaad41" member_type: VOTER } } I0507 06:33:53.327234 15598 sys_catalog.cc:340] T 00000000000000000000000000000000 P 96510d60d2b748329ce1e8d69faaad41 [sys.catalog]: This master's current role is: LEADER I0507 06:33:53.326714 15597 sys_catalog.cc:337] T 00000000000000000000000000000000 P 96510d60d2b748329ce1e8d69faaad41 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "96510d60d2b748329ce1e8d69faaad41" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96510d60d2b748329ce1e8d69faaad41" member_type: VOTER } } I0507 06:33:53.327515 15597 sys_catalog.cc:340] T 00000000000000000000000000000000 P 96510d60d2b748329ce1e8d69faaad41 [sys.catalog]: This master's current role is: LEADER I0507 06:33:53.329113 15593 sys_catalog.cc:424] T 00000000000000000000000000000000 P 96510d60d2b748329ce1e8d69faaad41 [sys.catalog]: configured and running, proceeding with master startup. I0507 06:33:53.329319 15600 catalog_manager.cc:1055] Loading table and tablet metadata into memory... I0507 06:33:53.333699 15600 catalog_manager.cc:1066] Initializing Kudu internal certificate authority... I0507 06:33:53.378549 15600 catalog_manager.cc:950] Generated new certificate authority record I0507 06:33:53.379981 15600 catalog_manager.cc:1075] Loading token signing keys... I0507 06:33:53.387339 15600 catalog_manager.cc:4292] T 00000000000000000000000000000000 P 96510d60d2b748329ce1e8d69faaad41: Generated new TSK 0 WARNING: Logging before InitGoogleLogging() is written to STDERR W0507 06:33:53.459879 15595 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0507 06:33:53.460086 15595 flags.cc:404] Enabled unsafe flag: --max_cell_size_bytes=1000000 W0507 06:33:53.460134 15595 flags.cc:404] Enabled unsafe flag: --never_fsync=true W0507 06:33:53.462272 15595 flags.cc:404] Enabled experimental flag: --log_compression_codec=none W0507 06:33:53.462332 15595 flags.cc:404] Enabled experimental flag: --log_max_segments_to_retain=3 W0507 06:33:53.462383 15595 flags.cc:404] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true W0507 06:33:53.462425 15595 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W0507 06:33:53.462517 15595 flags.cc:404] Enabled experimental flag: --log_target_replay_size_mb=1 W0507 06:33:53.462570 15595 flags.cc:404] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.57.193 I0507 06:33:53.466742 15595 tablet_server_main.cc:78] Tablet server non-default flags: --follower_unavailable_considered_failed_sec=5 --log_compression_codec=none --log_max_segments_to_retain=3 --log_cache_size_limit_mb=1 --log_async_preallocate_segments=false --log_segment_size_mb=1 --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_4.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_4.1557210500371863-231/raft_consensus-itest-cluster/ts-0/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.0.57.193:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_4.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data/info.pb --webserver_interface=127.0.57.193 --webserver_port=0 --max_cell_size_bytes=1000000 --tserver_master_addrs=127.0.57.254:34189 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --log_target_replay_size_mb=1 --maintenance_manager_polling_interval_ms=100 --enable_minidumps=false --local_ip_for_outbound_sockets=127.0.57.193 --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_4.1557210500371863-231/raft_consensus-itest-cluster/ts-0/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.10.0-cdh6.x-SNAPSHOT revision 7d0ee26697926e87af9ef395008799577f6ab43f build type FASTDEBUG built by jenkins at 06 May 2019 22:44:21 PST on quasar-knmpfx-1.vpc.cloudera.com build id 322017 ASAN enabled I0507 06:33:53.474014 15595 system_ntp.cc:202] NTP initialized. Skew: 500ppm Current error: 191016us I0507 06:33:53.474642 15595 fs_manager.cc:263] Metadata directory not provided I0507 06:33:53.474706 15595 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0507 06:33:53.474977 15595 server_base.cc:433] Could not load existing FS layout: Not found: could not find a healthy instance file I0507 06:33:53.475039 15595 server_base.cc:434] Attempting to create new FS layout instead I0507 06:33:53.477414 15595 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_4.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data/instance: uuid: "41cd5177c3884ec29a257f84325c090b" format_stamp: "Formatted at 2019-05-07 06:33:53 on dist-test-slave-dist-test-slave-rqp6" I0507 06:33:53.477957 15595 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_4.1557210500371863-231/raft_consensus-itest-cluster/ts-0/wal/instance: uuid: "41cd5177c3884ec29a257f84325c090b" format_stamp: "Formatted at 2019-05-07 06:33:53 on dist-test-slave-dist-test-slave-rqp6" I0507 06:33:53.490516 15595 fs_manager.cc:503] Time spent creating directory manager: real 0.012s user 0.004s sys 0.008s I0507 06:33:53.491662 15595 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0507 06:33:53.492276 15595 file_cache.cc:466] Constructed file cache lbm with capacity 419430 I0507 06:33:53.498636 15595 fs_manager.cc:419] Time spent opening block manager: real 0.005s user 0.004s sys 0.000s I0507 06:33:53.498749 15595 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_4.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_4.1557210500371863-231/raft_consensus-itest-cluster/ts-0/wal uuid: "41cd5177c3884ec29a257f84325c090b" format_stamp: "Formatted at 2019-05-07 06:33:53 on dist-test-slave-dist-test-slave-rqp6" I0507 06:33:53.498922 15595 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_4.1557210500371863-231/raft_consensus-itest-cluster/ts-0/wal metadata directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_4.1557210500371863-231/raft_consensus-itest-cluster/ts-0/wal 1 data directories: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_4.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0507 06:33:53.530120 15595 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0507 06:33:53.531707 15595 ts_tablet_manager.cc:356] Loaded tablet metadata (0 total tablets, 0 live tablets) I0507 06:33:53.531785 15595 ts_tablet_manager.cc:376] Registered 0 tablets I0507 06:33:53.635807 15595 rpc_server.cc:205] RPC server started. Bound to: 127.0.57.193:33409 I0507 06:33:53.636909 15595 webserver.cc:308] Webserver started at http://127.0.57.193:39269/ using document root and password file I0507 06:33:53.637567 15595 server_base.cc:608] Dumped server information to /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_4.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data/info.pb I0507 06:33:53.646384 231 external_mini_cluster.cc:995] Started /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver as pid 15595 I0507 06:33:53.647133 231 external_mini_cluster.cc:933] Running /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_4.1557210500371863-231/raft_consensus-itest-cluster/ts-1/wal --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_4.1557210500371863-231/raft_consensus-itest-cluster/ts-1/data --block_manager=log --rpc_bind_addresses=127.0.57.194:0 --local_ip_for_outbound_sockets=127.0.57.194 --webserver_interface=127.0.57.194 --webserver_port=0 --tserver_master_addrs=127.0.57.254:34189 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_4.1557210500371863-231/raft_consensus-itest-cluster/ts-1/logs --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_4.1557210500371863-231/raft_consensus-itest-cluster/ts-1/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=true --follower_unavailable_considered_failed_sec=5 --log_compression_codec=none --log_cache_size_limit_mb=1 --log_segment_size_mb=1 --log_async_preallocate_segments=false --log_min_segments_to_retain=1 --log_max_segments_to_retain=3 --maintenance_manager_polling_interval_ms=100 --log_target_replay_size_mb=1 --max_cell_size_bytes=1000000 with env {} I0507 06:33:53.653628 15723 heartbeater.cc:345] Connected to a master server at 127.0.57.254:34189 I0507 06:33:53.653838 15723 heartbeater.cc:426] Registering TS with master... I0507 06:33:53.654340 15723 heartbeater.cc:475] Master 127.0.57.254:34189 requested a full tablet report, sending... I0507 06:33:53.656075 15560 ts_manager.cc:159] Registered new tserver with Master: 41cd5177c3884ec29a257f84325c090b (127.0.57.193:33409) I0507 06:33:53.657861 15560 master_service.cc:273] Signed X509 certificate for tserver {username='slave'} at 127.0.57.193:49781 WARNING: Logging before InitGoogleLogging() is written to STDERR W0507 06:33:53.821768 15727 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0507 06:33:53.821996 15727 flags.cc:404] Enabled unsafe flag: --max_cell_size_bytes=1000000 W0507 06:33:53.822082 15727 flags.cc:404] Enabled unsafe flag: --never_fsync=true W0507 06:33:53.824110 15727 flags.cc:404] Enabled experimental flag: --log_compression_codec=none W0507 06:33:53.824172 15727 flags.cc:404] Enabled experimental flag: --log_max_segments_to_retain=3 W0507 06:33:53.824208 15727 flags.cc:404] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true W0507 06:33:53.824259 15727 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W0507 06:33:53.824348 15727 flags.cc:404] Enabled experimental flag: --log_target_replay_size_mb=1 W0507 06:33:53.824404 15727 flags.cc:404] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.57.194 I0507 06:33:53.828696 15727 tablet_server_main.cc:78] Tablet server non-default flags: --follower_unavailable_considered_failed_sec=5 --log_compression_codec=none --log_max_segments_to_retain=3 --log_cache_size_limit_mb=1 --log_async_preallocate_segments=false --log_segment_size_mb=1 --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_4.1557210500371863-231/raft_consensus-itest-cluster/ts-1/data --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_4.1557210500371863-231/raft_consensus-itest-cluster/ts-1/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.0.57.194:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_4.1557210500371863-231/raft_consensus-itest-cluster/ts-1/data/info.pb --webserver_interface=127.0.57.194 --webserver_port=0 --max_cell_size_bytes=1000000 --tserver_master_addrs=127.0.57.254:34189 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --log_target_replay_size_mb=1 --maintenance_manager_polling_interval_ms=100 --enable_minidumps=false --local_ip_for_outbound_sockets=127.0.57.194 --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_4.1557210500371863-231/raft_consensus-itest-cluster/ts-1/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.10.0-cdh6.x-SNAPSHOT revision 7d0ee26697926e87af9ef395008799577f6ab43f build type FASTDEBUG built by jenkins at 06 May 2019 22:44:21 PST on quasar-knmpfx-1.vpc.cloudera.com build id 322017 ASAN enabled I0507 06:33:53.836861 15727 system_ntp.cc:202] NTP initialized. Skew: 500ppm Current error: 191016us I0507 06:33:53.837572 15727 fs_manager.cc:263] Metadata directory not provided I0507 06:33:53.837646 15727 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0507 06:33:53.837934 15727 server_base.cc:433] Could not load existing FS layout: Not found: could not find a healthy instance file I0507 06:33:53.838001 15727 server_base.cc:434] Attempting to create new FS layout instead I0507 06:33:53.840601 15727 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_4.1557210500371863-231/raft_consensus-itest-cluster/ts-1/data/instance: uuid: "407e162251194eb69b048b59362a51bf" format_stamp: "Formatted at 2019-05-07 06:33:53 on dist-test-slave-dist-test-slave-rqp6" I0507 06:33:53.841174 15727 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_4.1557210500371863-231/raft_consensus-itest-cluster/ts-1/wal/instance: uuid: "407e162251194eb69b048b59362a51bf" format_stamp: "Formatted at 2019-05-07 06:33:53 on dist-test-slave-dist-test-slave-rqp6" I0507 06:33:53.848002 15727 fs_manager.cc:503] Time spent creating directory manager: real 0.007s user 0.005s sys 0.001s I0507 06:33:53.849264 15727 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0507 06:33:53.849872 15727 file_cache.cc:466] Constructed file cache lbm with capacity 419430 I0507 06:33:53.853282 15727 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s I0507 06:33:53.853374 15727 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_4.1557210500371863-231/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_4.1557210500371863-231/raft_consensus-itest-cluster/ts-1/wal uuid: "407e162251194eb69b048b59362a51bf" format_stamp: "Formatted at 2019-05-07 06:33:53 on dist-test-slave-dist-test-slave-rqp6" I0507 06:33:53.853536 15727 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_4.1557210500371863-231/raft_consensus-itest-cluster/ts-1/wal metadata directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_4.1557210500371863-231/raft_consensus-itest-cluster/ts-1/wal 1 data directories: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_4.1557210500371863-231/raft_consensus-itest-cluster/ts-1/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0507 06:33:53.903959 15727 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0507 06:33:53.905809 15727 ts_tablet_manager.cc:356] Loaded tablet metadata (0 total tablets, 0 live tablets) I0507 06:33:53.905894 15727 ts_tablet_manager.cc:376] Registered 0 tablets I0507 06:33:53.931493 15727 rpc_server.cc:205] RPC server started. Bound to: 127.0.57.194:32837 I0507 06:33:53.932819 15727 webserver.cc:308] Webserver started at http://127.0.57.194:44291/ using document root and password file I0507 06:33:53.933579 15727 server_base.cc:608] Dumped server information to /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_4.1557210500371863-231/raft_consensus-itest-cluster/ts-1/data/info.pb I0507 06:33:53.942463 231 external_mini_cluster.cc:995] Started /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver as pid 15727 I0507 06:33:53.943147 231 external_mini_cluster.cc:933] Running /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_4.1557210500371863-231/raft_consensus-itest-cluster/ts-2/wal --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_4.1557210500371863-231/raft_consensus-itest-cluster/ts-2/data --block_manager=log --rpc_bind_addresses=127.0.57.195:0 --local_ip_for_outbound_sockets=127.0.57.195 --webserver_interface=127.0.57.195 --webserver_port=0 --tserver_master_addrs=127.0.57.254:34189 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_4.1557210500371863-231/raft_consensus-itest-cluster/ts-2/logs --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_4.1557210500371863-231/raft_consensus-itest-cluster/ts-2/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=true --follower_unavailable_considered_failed_sec=5 --log_compression_codec=none --log_cache_size_limit_mb=1 --log_segment_size_mb=1 --log_async_preallocate_segments=false --log_min_segments_to_retain=1 --log_max_segments_to_retain=3 --maintenance_manager_polling_interval_ms=100 --log_target_replay_size_mb=1 --max_cell_size_bytes=1000000 with env {} I0507 06:33:53.947633 15846 heartbeater.cc:345] Connected to a master server at 127.0.57.254:34189 I0507 06:33:53.947868 15846 heartbeater.cc:426] Registering TS with master... I0507 06:33:53.948439 15846 heartbeater.cc:475] Master 127.0.57.254:34189 requested a full tablet report, sending... I0507 06:33:53.949679 15560 ts_manager.cc:159] Registered new tserver with Master: 407e162251194eb69b048b59362a51bf (127.0.57.194:32837) I0507 06:33:53.950673 15560 master_service.cc:273] Signed X509 certificate for tserver {username='slave'} at 127.0.57.194:39315 WARNING: Logging before InitGoogleLogging() is written to STDERR W0507 06:33:54.119112 15849 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0507 06:33:54.119340 15849 flags.cc:404] Enabled unsafe flag: --max_cell_size_bytes=1000000 W0507 06:33:54.119387 15849 flags.cc:404] Enabled unsafe flag: --never_fsync=true W0507 06:33:54.121533 15849 flags.cc:404] Enabled experimental flag: --log_compression_codec=none W0507 06:33:54.121596 15849 flags.cc:404] Enabled experimental flag: --log_max_segments_to_retain=3 W0507 06:33:54.121655 15849 flags.cc:404] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true W0507 06:33:54.121707 15849 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W0507 06:33:54.121809 15849 flags.cc:404] Enabled experimental flag: --log_target_replay_size_mb=1 W0507 06:33:54.121867 15849 flags.cc:404] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.57.195 I0507 06:33:54.126049 15849 tablet_server_main.cc:78] Tablet server non-default flags: --follower_unavailable_considered_failed_sec=5 --log_compression_codec=none --log_max_segments_to_retain=3 --log_cache_size_limit_mb=1 --log_async_preallocate_segments=false --log_segment_size_mb=1 --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_4.1557210500371863-231/raft_consensus-itest-cluster/ts-2/data --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_4.1557210500371863-231/raft_consensus-itest-cluster/ts-2/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.0.57.195:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_4.1557210500371863-231/raft_consensus-itest-cluster/ts-2/data/info.pb --webserver_interface=127.0.57.195 --webserver_port=0 --max_cell_size_bytes=1000000 --tserver_master_addrs=127.0.57.254:34189 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --log_target_replay_size_mb=1 --maintenance_manager_polling_interval_ms=100 --enable_minidumps=false --local_ip_for_outbound_sockets=127.0.57.195 --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_4.1557210500371863-231/raft_consensus-itest-cluster/ts-2/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.10.0-cdh6.x-SNAPSHOT revision 7d0ee26697926e87af9ef395008799577f6ab43f build type FASTDEBUG built by jenkins at 06 May 2019 22:44:21 PST on quasar-knmpfx-1.vpc.cloudera.com build id 322017 ASAN enabled I0507 06:33:54.134241 15849 system_ntp.cc:202] NTP initialized. Skew: 500ppm Current error: 191516us I0507 06:33:54.134956 15849 fs_manager.cc:263] Metadata directory not provided I0507 06:33:54.135025 15849 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0507 06:33:54.135344 15849 server_base.cc:433] Could not load existing FS layout: Not found: could not find a healthy instance file I0507 06:33:54.135413 15849 server_base.cc:434] Attempting to create new FS layout instead I0507 06:33:54.138450 15849 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_4.1557210500371863-231/raft_consensus-itest-cluster/ts-2/data/instance: uuid: "1b055b8db383436b94d48378a89755a8" format_stamp: "Formatted at 2019-05-07 06:33:54 on dist-test-slave-dist-test-slave-rqp6" I0507 06:33:54.139050 15849 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_4.1557210500371863-231/raft_consensus-itest-cluster/ts-2/wal/instance: uuid: "1b055b8db383436b94d48378a89755a8" format_stamp: "Formatted at 2019-05-07 06:33:54 on dist-test-slave-dist-test-slave-rqp6" I0507 06:33:54.146483 15849 fs_manager.cc:503] Time spent creating directory manager: real 0.007s user 0.002s sys 0.005s I0507 06:33:54.147692 15849 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0507 06:33:54.148361 15849 file_cache.cc:466] Constructed file cache lbm with capacity 419430 I0507 06:33:54.151943 15849 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.000s sys 0.000s I0507 06:33:54.152040 15849 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_4.1557210500371863-231/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_4.1557210500371863-231/raft_consensus-itest-cluster/ts-2/wal uuid: "1b055b8db383436b94d48378a89755a8" format_stamp: "Formatted at 2019-05-07 06:33:54 on dist-test-slave-dist-test-slave-rqp6" I0507 06:33:54.152225 15849 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_4.1557210500371863-231/raft_consensus-itest-cluster/ts-2/wal metadata directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_4.1557210500371863-231/raft_consensus-itest-cluster/ts-2/wal 1 data directories: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_4.1557210500371863-231/raft_consensus-itest-cluster/ts-2/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0507 06:33:54.183789 15849 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0507 06:33:54.185667 15849 ts_tablet_manager.cc:356] Loaded tablet metadata (0 total tablets, 0 live tablets) I0507 06:33:54.185762 15849 ts_tablet_manager.cc:376] Registered 0 tablets I0507 06:33:54.214911 15849 rpc_server.cc:205] RPC server started. Bound to: 127.0.57.195:37465 I0507 06:33:54.216130 15849 webserver.cc:308] Webserver started at http://127.0.57.195:38165/ using document root and password file I0507 06:33:54.216842 15849 server_base.cc:608] Dumped server information to /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_4.1557210500371863-231/raft_consensus-itest-cluster/ts-2/data/info.pb I0507 06:33:54.219441 231 external_mini_cluster.cc:995] Started /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver as pid 15849 I0507 06:33:54.230537 15968 heartbeater.cc:345] Connected to a master server at 127.0.57.254:34189 I0507 06:33:54.230806 15968 heartbeater.cc:426] Registering TS with master... I0507 06:33:54.231359 15968 heartbeater.cc:475] Master 127.0.57.254:34189 requested a full tablet report, sending... I0507 06:33:54.232414 15559 ts_manager.cc:159] Registered new tserver with Master: 1b055b8db383436b94d48378a89755a8 (127.0.57.195:37465) I0507 06:33:54.233366 15559 master_service.cc:273] Signed X509 certificate for tserver {username='slave'} at 127.0.57.195:42321 I0507 06:33:54.235132 231 external_mini_cluster.cc:623] 3 TS(s) registered with all masters I0507 06:33:54.246290 15559 catalog_manager.cc:1415] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37678: name: "TestTable" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 3 split_rows_range_bounds { } partition_schema { range_schema { columns { name: "key" } } } W0507 06:33:54.250903 15559 catalog_manager.cc:1574] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestTable in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers. I0507 06:33:54.308020 15905 tablet_service.cc:1016] Processing CreateTablet for tablet fdc69131212841d285ea88aadf8b6bd1 (table=TestTable [id=a9924983367f4c9ea12957ce08a021a0]), partition=RANGE (key) PARTITION UNBOUNDED I0507 06:33:54.308965 15905 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet fdc69131212841d285ea88aadf8b6bd1. 1 dirs total, 0 dirs full, 0 dirs failed I0507 06:33:54.310513 15783 tablet_service.cc:1016] Processing CreateTablet for tablet fdc69131212841d285ea88aadf8b6bd1 (table=TestTable [id=a9924983367f4c9ea12957ce08a021a0]), partition=RANGE (key) PARTITION UNBOUNDED I0507 06:33:54.311559 15783 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet fdc69131212841d285ea88aadf8b6bd1. 1 dirs total, 0 dirs full, 0 dirs failed I0507 06:33:54.313320 15660 tablet_service.cc:1016] Processing CreateTablet for tablet fdc69131212841d285ea88aadf8b6bd1 (table=TestTable [id=a9924983367f4c9ea12957ce08a021a0]), partition=RANGE (key) PARTITION UNBOUNDED I0507 06:33:54.314131 15660 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet fdc69131212841d285ea88aadf8b6bd1. 1 dirs total, 0 dirs full, 0 dirs failed I0507 06:33:54.318403 15987 tablet_bootstrap.cc:439] T fdc69131212841d285ea88aadf8b6bd1 P 1b055b8db383436b94d48378a89755a8: Bootstrap starting. I0507 06:33:54.320653 15987 tablet_bootstrap.cc:590] T fdc69131212841d285ea88aadf8b6bd1 P 1b055b8db383436b94d48378a89755a8: No blocks or log segments found. Creating new log. I0507 06:33:54.321516 15987 log.cc:525] T fdc69131212841d285ea88aadf8b6bd1 P 1b055b8db383436b94d48378a89755a8: Log is configured to *not* fsync() on all Append() calls I0507 06:33:54.322021 15988 tablet_bootstrap.cc:439] T fdc69131212841d285ea88aadf8b6bd1 P 407e162251194eb69b048b59362a51bf: Bootstrap starting. I0507 06:33:54.324059 15987 tablet_bootstrap.cc:439] T fdc69131212841d285ea88aadf8b6bd1 P 1b055b8db383436b94d48378a89755a8: No bootstrap required, opened a new log I0507 06:33:54.324286 15987 ts_tablet_manager.cc:1053] T fdc69131212841d285ea88aadf8b6bd1 P 1b055b8db383436b94d48378a89755a8: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s I0507 06:33:54.325232 15990 tablet_bootstrap.cc:439] T fdc69131212841d285ea88aadf8b6bd1 P 41cd5177c3884ec29a257f84325c090b: Bootstrap starting. I0507 06:33:54.327533 15990 tablet_bootstrap.cc:590] T fdc69131212841d285ea88aadf8b6bd1 P 41cd5177c3884ec29a257f84325c090b: No blocks or log segments found. Creating new log. I0507 06:33:54.329507 15988 tablet_bootstrap.cc:590] T fdc69131212841d285ea88aadf8b6bd1 P 407e162251194eb69b048b59362a51bf: No blocks or log segments found. Creating new log. I0507 06:33:54.329876 15987 raft_consensus.cc:340] T fdc69131212841d285ea88aadf8b6bd1 P 1b055b8db383436b94d48378a89755a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b055b8db383436b94d48378a89755a8" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 37465 } } peers { permanent_uuid: "41cd5177c3884ec29a257f84325c090b" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 33409 } } peers { permanent_uuid: "407e162251194eb69b048b59362a51bf" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 32837 } } I0507 06:33:54.330216 15987 raft_consensus.cc:366] T fdc69131212841d285ea88aadf8b6bd1 P 1b055b8db383436b94d48378a89755a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0507 06:33:54.330273 15987 raft_consensus.cc:705] T fdc69131212841d285ea88aadf8b6bd1 P 1b055b8db383436b94d48378a89755a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b055b8db383436b94d48378a89755a8, State: Initialized, Role: FOLLOWER I0507 06:33:54.330366 15988 log.cc:525] T fdc69131212841d285ea88aadf8b6bd1 P 407e162251194eb69b048b59362a51bf: Log is configured to *not* fsync() on all Append() calls I0507 06:33:54.330654 15987 consensus_queue.cc:229] T fdc69131212841d285ea88aadf8b6bd1 P 1b055b8db383436b94d48378a89755a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b055b8db383436b94d48378a89755a8" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 37465 } } peers { permanent_uuid: "41cd5177c3884ec29a257f84325c090b" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 33409 } } peers { permanent_uuid: "407e162251194eb69b048b59362a51bf" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 32837 } } I0507 06:33:54.330782 15990 log.cc:525] T fdc69131212841d285ea88aadf8b6bd1 P 41cd5177c3884ec29a257f84325c090b: Log is configured to *not* fsync() on all Append() calls I0507 06:33:54.331403 15968 heartbeater.cc:467] Master 127.0.57.254:34189 was elected leader, sending a full tablet report... I0507 06:33:54.331851 15987 ts_tablet_manager.cc:1081] T fdc69131212841d285ea88aadf8b6bd1 P 1b055b8db383436b94d48378a89755a8: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s I0507 06:33:54.332454 15988 tablet_bootstrap.cc:439] T fdc69131212841d285ea88aadf8b6bd1 P 407e162251194eb69b048b59362a51bf: No bootstrap required, opened a new log I0507 06:33:54.332747 15988 ts_tablet_manager.cc:1053] T fdc69131212841d285ea88aadf8b6bd1 P 407e162251194eb69b048b59362a51bf: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.000s I0507 06:33:54.333024 15990 tablet_bootstrap.cc:439] T fdc69131212841d285ea88aadf8b6bd1 P 41cd5177c3884ec29a257f84325c090b: No bootstrap required, opened a new log I0507 06:33:54.333293 15990 ts_tablet_manager.cc:1053] T fdc69131212841d285ea88aadf8b6bd1 P 41cd5177c3884ec29a257f84325c090b: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s I0507 06:33:54.338456 15988 raft_consensus.cc:340] T fdc69131212841d285ea88aadf8b6bd1 P 407e162251194eb69b048b59362a51bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b055b8db383436b94d48378a89755a8" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 37465 } } peers { permanent_uuid: "41cd5177c3884ec29a257f84325c090b" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 33409 } } peers { permanent_uuid: "407e162251194eb69b048b59362a51bf" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 32837 } } I0507 06:33:54.338827 15988 raft_consensus.cc:366] T fdc69131212841d285ea88aadf8b6bd1 P 407e162251194eb69b048b59362a51bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0507 06:33:54.338886 15988 raft_consensus.cc:705] T fdc69131212841d285ea88aadf8b6bd1 P 407e162251194eb69b048b59362a51bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 407e162251194eb69b048b59362a51bf, State: Initialized, Role: FOLLOWER I0507 06:33:54.338718 15990 raft_consensus.cc:340] T fdc69131212841d285ea88aadf8b6bd1 P 41cd5177c3884ec29a257f84325c090b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b055b8db383436b94d48378a89755a8" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 37465 } } peers { permanent_uuid: "41cd5177c3884ec29a257f84325c090b" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 33409 } } peers { permanent_uuid: "407e162251194eb69b048b59362a51bf" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 32837 } } I0507 06:33:54.339040 15990 raft_consensus.cc:366] T fdc69131212841d285ea88aadf8b6bd1 P 41cd5177c3884ec29a257f84325c090b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0507 06:33:54.339100 15990 raft_consensus.cc:705] T fdc69131212841d285ea88aadf8b6bd1 P 41cd5177c3884ec29a257f84325c090b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41cd5177c3884ec29a257f84325c090b, State: Initialized, Role: FOLLOWER I0507 06:33:54.339359 15988 consensus_queue.cc:229] T fdc69131212841d285ea88aadf8b6bd1 P 407e162251194eb69b048b59362a51bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b055b8db383436b94d48378a89755a8" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 37465 } } peers { permanent_uuid: "41cd5177c3884ec29a257f84325c090b" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 33409 } } peers { permanent_uuid: "407e162251194eb69b048b59362a51bf" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 32837 } } I0507 06:33:54.339488 15990 consensus_queue.cc:229] T fdc69131212841d285ea88aadf8b6bd1 P 41cd5177c3884ec29a257f84325c090b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b055b8db383436b94d48378a89755a8" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 37465 } } peers { permanent_uuid: "41cd5177c3884ec29a257f84325c090b" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 33409 } } peers { permanent_uuid: "407e162251194eb69b048b59362a51bf" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 32837 } } I0507 06:33:54.340319 15846 heartbeater.cc:467] Master 127.0.57.254:34189 was elected leader, sending a full tablet report... I0507 06:33:54.340754 15988 ts_tablet_manager.cc:1081] T fdc69131212841d285ea88aadf8b6bd1 P 407e162251194eb69b048b59362a51bf: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s I0507 06:33:54.340747 15723 heartbeater.cc:467] Master 127.0.57.254:34189 was elected leader, sending a full tablet report... I0507 06:33:54.341173 15990 ts_tablet_manager.cc:1081] T fdc69131212841d285ea88aadf8b6bd1 P 41cd5177c3884ec29a257f84325c090b: Time spent starting tablet: real 0.008s user 0.004s sys 0.004s I0507 06:33:54.344781 15994 raft_consensus.cc:472] T fdc69131212841d285ea88aadf8b6bd1 P 407e162251194eb69b048b59362a51bf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0507 06:33:54.344965 15994 raft_consensus.cc:494] T fdc69131212841d285ea88aadf8b6bd1 P 407e162251194eb69b048b59362a51bf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b055b8db383436b94d48378a89755a8" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 37465 } } peers { permanent_uuid: "41cd5177c3884ec29a257f84325c090b" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 33409 } } peers { permanent_uuid: "407e162251194eb69b048b59362a51bf" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 32837 } } I0507 06:33:54.346159 15994 leader_election.cc:296] T fdc69131212841d285ea88aadf8b6bd1 P 407e162251194eb69b048b59362a51bf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1b055b8db383436b94d48378a89755a8 (127.0.57.195:37465), 41cd5177c3884ec29a257f84325c090b (127.0.57.193:33409) I0507 06:33:54.351488 15680 tablet_service.cc:1286] Received RequestConsensusVote() RPC: tablet_id: "fdc69131212841d285ea88aadf8b6bd1" candidate_uuid: "407e162251194eb69b048b59362a51bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41cd5177c3884ec29a257f84325c090b" is_pre_election: true I0507 06:33:54.351857 15680 raft_consensus.cc:2346] T fdc69131212841d285ea88aadf8b6bd1 P 41cd5177c3884ec29a257f84325c090b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 407e162251194eb69b048b59362a51bf in term 0. I0507 06:33:54.352342 15925 tablet_service.cc:1286] Received RequestConsensusVote() RPC: tablet_id: "fdc69131212841d285ea88aadf8b6bd1" candidate_uuid: "407e162251194eb69b048b59362a51bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b055b8db383436b94d48378a89755a8" is_pre_election: true I0507 06:33:54.352460 15734 leader_election.cc:310] T fdc69131212841d285ea88aadf8b6bd1 P 407e162251194eb69b048b59362a51bf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 407e162251194eb69b048b59362a51bf, 41cd5177c3884ec29a257f84325c090b; no voters: I0507 06:33:54.352689 15925 raft_consensus.cc:2346] T fdc69131212841d285ea88aadf8b6bd1 P 1b055b8db383436b94d48378a89755a8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 407e162251194eb69b048b59362a51bf in term 0. I0507 06:33:54.352954 15994 raft_consensus.cc:2640] T fdc69131212841d285ea88aadf8b6bd1 P 407e162251194eb69b048b59362a51bf [term 0 FOLLOWER]: Leader pre-election won for term 1 I0507 06:33:54.353060 15994 raft_consensus.cc:472] T fdc69131212841d285ea88aadf8b6bd1 P 407e162251194eb69b048b59362a51bf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0507 06:33:54.353107 15994 raft_consensus.cc:2886] T fdc69131212841d285ea88aadf8b6bd1 P 407e162251194eb69b048b59362a51bf [term 0 FOLLOWER]: Advancing to term 1 I0507 06:33:54.355026 15994 raft_consensus.cc:494] T fdc69131212841d285ea88aadf8b6bd1 P 407e162251194eb69b048b59362a51bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b055b8db383436b94d48378a89755a8" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 37465 } } peers { permanent_uuid: "41cd5177c3884ec29a257f84325c090b" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 33409 } } peers { permanent_uuid: "407e162251194eb69b048b59362a51bf" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 32837 } } I0507 06:33:54.355568 15994 leader_election.cc:296] T fdc69131212841d285ea88aadf8b6bd1 P 407e162251194eb69b048b59362a51bf [CANDIDATE]: Term 1 election: Requested vote from peers 1b055b8db383436b94d48378a89755a8 (127.0.57.195:37465), 41cd5177c3884ec29a257f84325c090b (127.0.57.193:33409) I0507 06:33:54.356003 15925 tablet_service.cc:1286] Received RequestConsensusVote() RPC: tablet_id: "fdc69131212841d285ea88aadf8b6bd1" candidate_uuid: "407e162251194eb69b048b59362a51bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b055b8db383436b94d48378a89755a8" I0507 06:33:54.356112 15680 tablet_service.cc:1286] Received RequestConsensusVote() RPC: tablet_id: "fdc69131212841d285ea88aadf8b6bd1" candidate_uuid: "407e162251194eb69b048b59362a51bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41cd5177c3884ec29a257f84325c090b" I0507 06:33:54.356161 15925 raft_consensus.cc:2886] T fdc69131212841d285ea88aadf8b6bd1 P 1b055b8db383436b94d48378a89755a8 [term 0 FOLLOWER]: Advancing to term 1 I0507 06:33:54.356276 15680 raft_consensus.cc:2886] T fdc69131212841d285ea88aadf8b6bd1 P 41cd5177c3884ec29a257f84325c090b [term 0 FOLLOWER]: Advancing to term 1 I0507 06:33:54.358037 15925 raft_consensus.cc:2346] T fdc69131212841d285ea88aadf8b6bd1 P 1b055b8db383436b94d48378a89755a8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 407e162251194eb69b048b59362a51bf in term 1. I0507 06:33:54.358098 15680 raft_consensus.cc:2346] T fdc69131212841d285ea88aadf8b6bd1 P 41cd5177c3884ec29a257f84325c090b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 407e162251194eb69b048b59362a51bf in term 1. I0507 06:33:54.358441 15735 leader_election.cc:310] T fdc69131212841d285ea88aadf8b6bd1 P 407e162251194eb69b048b59362a51bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b055b8db383436b94d48378a89755a8, 407e162251194eb69b048b59362a51bf; no voters: I0507 06:33:54.358765 15994 raft_consensus.cc:2640] T fdc69131212841d285ea88aadf8b6bd1 P 407e162251194eb69b048b59362a51bf [term 1 FOLLOWER]: Leader election won for term 1 I0507 06:33:54.359119 15994 raft_consensus.cc:667] T fdc69131212841d285ea88aadf8b6bd1 P 407e162251194eb69b048b59362a51bf [term 1 LEADER]: Becoming Leader. State: Replica: 407e162251194eb69b048b59362a51bf, State: Running, Role: LEADER I0507 06:33:54.359549 15994 consensus_queue.cc:206] T fdc69131212841d285ea88aadf8b6bd1 P 407e162251194eb69b048b59362a51bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b055b8db383436b94d48378a89755a8" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 37465 } } peers { permanent_uuid: "41cd5177c3884ec29a257f84325c090b" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 33409 } } peers { permanent_uuid: "407e162251194eb69b048b59362a51bf" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 32837 } } I0507 06:33:54.363294 15559 catalog_manager.cc:3988] T fdc69131212841d285ea88aadf8b6bd1 P 407e162251194eb69b048b59362a51bf reported cstate change: term changed from 0 to 1, leader changed from to 407e162251194eb69b048b59362a51bf (127.0.57.194). New cstate: current_term: 1 leader_uuid: "407e162251194eb69b048b59362a51bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b055b8db383436b94d48378a89755a8" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 37465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "41cd5177c3884ec29a257f84325c090b" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 33409 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "407e162251194eb69b048b59362a51bf" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 32837 } health_report { overall_health: HEALTHY } } } I0507 06:33:54.391654 231 external_mini_cluster.cc:623] 3 TS(s) registered with all masters I0507 06:33:54.393110 231 ts_itest-base.cc:232] Waiting for 1 tablets on tserver 41cd5177c3884ec29a257f84325c090b to finish bootstrapping I0507 06:33:54.399369 231 ts_itest-base.cc:232] Waiting for 1 tablets on tserver 407e162251194eb69b048b59362a51bf to finish bootstrapping I0507 06:33:54.405941 231 ts_itest-base.cc:232] Waiting for 1 tablets on tserver 1b055b8db383436b94d48378a89755a8 to finish bootstrapping W0507 06:33:54.418947 15969 tablet.cc:1786] T fdc69131212841d285ea88aadf8b6bd1 P 1b055b8db383436b94d48378a89755a8: Can't schedule compaction. Clean time has not been advanced past its initial value. I0507 06:33:54.426578 231 cluster_itest_util.cc:258] Not converged past 1 yet: 0.0 1.1 0.0 W0507 06:33:54.436126 15847 tablet.cc:1786] T fdc69131212841d285ea88aadf8b6bd1 P 407e162251194eb69b048b59362a51bf: Can't schedule compaction. Clean time has not been advanced past its initial value. W0507 06:33:54.440273 15724 tablet.cc:1786] T fdc69131212841d285ea88aadf8b6bd1 P 41cd5177c3884ec29a257f84325c090b: Can't schedule compaction. Clean time has not been advanced past its initial value. I0507 06:33:54.531006 231 cluster_itest_util.cc:258] Not converged past 1 yet: 0.0 1.1 0.0 I0507 06:33:54.735357 231 cluster_itest_util.cc:258] Not converged past 1 yet: 0.0 1.1 0.0 I0507 06:33:54.752777 15925 raft_consensus.cc:1181] T fdc69131212841d285ea88aadf8b6bd1 P 1b055b8db383436b94d48378a89755a8 [term 1 FOLLOWER]: Refusing update from remote peer 407e162251194eb69b048b59362a51bf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch) I0507 06:33:54.753634 15994 consensus_queue.cc:984] T fdc69131212841d285ea88aadf8b6bd1 P 407e162251194eb69b048b59362a51bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b055b8db383436b94d48378a89755a8" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 37465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0507 06:33:54.759958 15680 raft_consensus.cc:1181] T fdc69131212841d285ea88aadf8b6bd1 P 41cd5177c3884ec29a257f84325c090b [term 1 FOLLOWER]: Refusing update from remote peer 407e162251194eb69b048b59362a51bf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch) I0507 06:33:54.761006 16003 consensus_queue.cc:984] T fdc69131212841d285ea88aadf8b6bd1 P 407e162251194eb69b048b59362a51bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "41cd5177c3884ec29a257f84325c090b" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 33409 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0507 06:33:55.039881 231 external_mini_cluster.cc:1172] Attempting to check leaks for /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver pid 15727 I0507 06:33:55.040843 15843 generic_service.cc:171] Checking for leaks (request via RPC) I0507 06:33:55.094971 231 external_mini_cluster.cc:1131] Killing /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver with pid 15727 I0507 06:33:56.259745 16008 raft_consensus.cc:472] T fdc69131212841d285ea88aadf8b6bd1 P 1b055b8db383436b94d48378a89755a8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 407e162251194eb69b048b59362a51bf) I0507 06:33:56.259910 16008 raft_consensus.cc:494] T fdc69131212841d285ea88aadf8b6bd1 P 1b055b8db383436b94d48378a89755a8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b055b8db383436b94d48378a89755a8" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 37465 } } peers { permanent_uuid: "41cd5177c3884ec29a257f84325c090b" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 33409 } } peers { permanent_uuid: "407e162251194eb69b048b59362a51bf" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 32837 } } I0507 06:33:56.261121 16008 leader_election.cc:296] T fdc69131212841d285ea88aadf8b6bd1 P 1b055b8db383436b94d48378a89755a8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 41cd5177c3884ec29a257f84325c090b (127.0.57.193:33409), 407e162251194eb69b048b59362a51bf (127.0.57.194:32837) I0507 06:33:56.263370 16012 raft_consensus.cc:472] T fdc69131212841d285ea88aadf8b6bd1 P 41cd5177c3884ec29a257f84325c090b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 407e162251194eb69b048b59362a51bf) I0507 06:33:56.263833 16012 raft_consensus.cc:494] T fdc69131212841d285ea88aadf8b6bd1 P 41cd5177c3884ec29a257f84325c090b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b055b8db383436b94d48378a89755a8" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 37465 } } peers { permanent_uuid: "41cd5177c3884ec29a257f84325c090b" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 33409 } } peers { permanent_uuid: "407e162251194eb69b048b59362a51bf" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 32837 } } W0507 06:33:56.264492 15859 leader_election.cc:341] T fdc69131212841d285ea88aadf8b6bd1 P 1b055b8db383436b94d48378a89755a8 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 407e162251194eb69b048b59362a51bf (127.0.57.194:32837): Network error: Client connection negotiation failed: client connection to 127.0.57.194:32837: connect: Connection refused (error 111) I0507 06:33:56.266530 16012 leader_election.cc:296] T fdc69131212841d285ea88aadf8b6bd1 P 41cd5177c3884ec29a257f84325c090b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1b055b8db383436b94d48378a89755a8 (127.0.57.195:37465), 407e162251194eb69b048b59362a51bf (127.0.57.194:32837) W0507 06:33:56.266973 15614 leader_election.cc:341] T fdc69131212841d285ea88aadf8b6bd1 P 41cd5177c3884ec29a257f84325c090b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 407e162251194eb69b048b59362a51bf (127.0.57.194:32837): Network error: Client connection negotiation failed: client connection to 127.0.57.194:32837: connect: Connection refused (error 111) I0507 06:33:56.270285 15680 tablet_service.cc:1286] Received RequestConsensusVote() RPC: tablet_id: "fdc69131212841d285ea88aadf8b6bd1" candidate_uuid: "1b055b8db383436b94d48378a89755a8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "41cd5177c3884ec29a257f84325c090b" is_pre_election: true I0507 06:33:56.270517 15680 raft_consensus.cc:2346] T fdc69131212841d285ea88aadf8b6bd1 P 41cd5177c3884ec29a257f84325c090b [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b055b8db383436b94d48378a89755a8 in term 1. I0507 06:33:56.270918 15856 leader_election.cc:310] T fdc69131212841d285ea88aadf8b6bd1 P 1b055b8db383436b94d48378a89755a8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1b055b8db383436b94d48378a89755a8, 41cd5177c3884ec29a257f84325c090b; no voters: 407e162251194eb69b048b59362a51bf I0507 06:33:56.271420 16008 raft_consensus.cc:2640] T fdc69131212841d285ea88aadf8b6bd1 P 1b055b8db383436b94d48378a89755a8 [term 1 FOLLOWER]: Leader pre-election won for term 2 I0507 06:33:56.271518 16008 raft_consensus.cc:472] T fdc69131212841d285ea88aadf8b6bd1 P 1b055b8db383436b94d48378a89755a8 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 407e162251194eb69b048b59362a51bf) I0507 06:33:56.271574 16008 raft_consensus.cc:2886] T fdc69131212841d285ea88aadf8b6bd1 P 1b055b8db383436b94d48378a89755a8 [term 1 FOLLOWER]: Advancing to term 2 I0507 06:33:56.271988 15925 tablet_service.cc:1286] Received RequestConsensusVote() RPC: tablet_id: "fdc69131212841d285ea88aadf8b6bd1" candidate_uuid: "41cd5177c3884ec29a257f84325c090b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1b055b8db383436b94d48378a89755a8" is_pre_election: true I0507 06:33:56.273403 16008 raft_consensus.cc:494] T fdc69131212841d285ea88aadf8b6bd1 P 1b055b8db383436b94d48378a89755a8 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b055b8db383436b94d48378a89755a8" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 37465 } } peers { permanent_uuid: "41cd5177c3884ec29a257f84325c090b" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 33409 } } peers { permanent_uuid: "407e162251194eb69b048b59362a51bf" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 32837 } } I0507 06:33:56.273751 15925 raft_consensus.cc:2276] T fdc69131212841d285ea88aadf8b6bd1 P 1b055b8db383436b94d48378a89755a8 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 41cd5177c3884ec29a257f84325c090b in current term 2: Already voted for candidate 1b055b8db383436b94d48378a89755a8 in this term. I0507 06:33:56.273941 16008 leader_election.cc:296] T fdc69131212841d285ea88aadf8b6bd1 P 1b055b8db383436b94d48378a89755a8 [CANDIDATE]: Term 2 election: Requested vote from peers 41cd5177c3884ec29a257f84325c090b (127.0.57.193:33409), 407e162251194eb69b048b59362a51bf (127.0.57.194:32837) I0507 06:33:56.274413 15680 tablet_service.cc:1286] Received RequestConsensusVote() RPC: tablet_id: "fdc69131212841d285ea88aadf8b6bd1" candidate_uuid: "1b055b8db383436b94d48378a89755a8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "41cd5177c3884ec29a257f84325c090b" I0507 06:33:56.274586 15680 raft_consensus.cc:2886] T fdc69131212841d285ea88aadf8b6bd1 P 41cd5177c3884ec29a257f84325c090b [term 1 FOLLOWER]: Advancing to term 2 I0507 06:33:56.274736 15612 leader_election.cc:310] T fdc69131212841d285ea88aadf8b6bd1 P 41cd5177c3884ec29a257f84325c090b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 41cd5177c3884ec29a257f84325c090b; no voters: 1b055b8db383436b94d48378a89755a8, 407e162251194eb69b048b59362a51bf I0507 06:33:56.276222 15680 raft_consensus.cc:2346] T fdc69131212841d285ea88aadf8b6bd1 P 41cd5177c3884ec29a257f84325c090b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b055b8db383436b94d48378a89755a8 in term 2. I0507 06:33:56.276381 16012 raft_consensus.cc:2588] T fdc69131212841d285ea88aadf8b6bd1 P 41cd5177c3884ec29a257f84325c090b [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority I0507 06:33:56.276726 15856 leader_election.cc:310] T fdc69131212841d285ea88aadf8b6bd1 P 1b055b8db383436b94d48378a89755a8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b055b8db383436b94d48378a89755a8, 41cd5177c3884ec29a257f84325c090b; no voters: W0507 06:33:56.276913 15859 leader_election.cc:341] T fdc69131212841d285ea88aadf8b6bd1 P 1b055b8db383436b94d48378a89755a8 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 407e162251194eb69b048b59362a51bf (127.0.57.194:32837): Network error: Client connection negotiation failed: client connection to 127.0.57.194:32837: connect: Connection refused (error 111) I0507 06:33:56.277137 16008 raft_consensus.cc:2640] T fdc69131212841d285ea88aadf8b6bd1 P 1b055b8db383436b94d48378a89755a8 [term 2 FOLLOWER]: Leader election won for term 2 I0507 06:33:56.277627 16008 raft_consensus.cc:667] T fdc69131212841d285ea88aadf8b6bd1 P 1b055b8db383436b94d48378a89755a8 [term 2 LEADER]: Becoming Leader. State: Replica: 1b055b8db383436b94d48378a89755a8, State: Running, Role: LEADER I0507 06:33:56.278009 16008 consensus_queue.cc:206] T fdc69131212841d285ea88aadf8b6bd1 P 1b055b8db383436b94d48378a89755a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b055b8db383436b94d48378a89755a8" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 37465 } } peers { permanent_uuid: "41cd5177c3884ec29a257f84325c090b" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 33409 } } peers { permanent_uuid: "407e162251194eb69b048b59362a51bf" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 32837 } } I0507 06:33:56.281023 15560 catalog_manager.cc:3988] T fdc69131212841d285ea88aadf8b6bd1 P 1b055b8db383436b94d48378a89755a8 reported cstate change: term changed from 1 to 2, leader changed from 407e162251194eb69b048b59362a51bf (127.0.57.194) to 1b055b8db383436b94d48378a89755a8 (127.0.57.195). New cstate: current_term: 2 leader_uuid: "1b055b8db383436b94d48378a89755a8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b055b8db383436b94d48378a89755a8" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 37465 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "41cd5177c3884ec29a257f84325c090b" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 33409 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "407e162251194eb69b048b59362a51bf" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 32837 } health_report { overall_health: UNKNOWN } } } I0507 06:33:56.645026 231 test_util.cc:212] Using random seed: 1724003463 I0507 06:33:56.662071 231 test_workload.cc:306] TestWorkload: Skipping table creation because table TestTable already exists I0507 06:33:56.664417 231 raft_consensus-itest-base.cc:271] Waiting until we've written at least 4MB... W0507 06:33:56.681211 15859 consensus_peers.cc:457] T fdc69131212841d285ea88aadf8b6bd1 P 1b055b8db383436b94d48378a89755a8 -> Peer 407e162251194eb69b048b59362a51bf (127.0.57.194:32837): Couldn't send request to peer 407e162251194eb69b048b59362a51bf. Status: Network error: Client connection negotiation failed: client connection to 127.0.57.194:32837: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry. I0507 06:33:56.685183 15680 raft_consensus.cc:1181] T fdc69131212841d285ea88aadf8b6bd1 P 41cd5177c3884ec29a257f84325c090b [term 2 FOLLOWER]: Refusing update from remote peer 1b055b8db383436b94d48378a89755a8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 3. (index mismatch) I0507 06:33:56.685948 16016 consensus_queue.cc:984] T fdc69131212841d285ea88aadf8b6bd1 P 1b055b8db383436b94d48378a89755a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "41cd5177c3884ec29a257f84325c090b" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 33409 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s I0507 06:33:56.748025 15724 maintenance_manager.cc:306] P 41cd5177c3884ec29a257f84325c090b: Scheduling FlushMRSOp(fdc69131212841d285ea88aadf8b6bd1): 1840774 bytes log retention I0507 06:33:56.807452 15618 maintenance_manager.cc:501] P 41cd5177c3884ec29a257f84325c090b: FlushMRSOp(fdc69131212841d285ea88aadf8b6bd1) complete. Timing: real 0.059s user 0.042s sys 0.004s Metrics: {"bytes_written":135869,"cfile_init":1,"compiler_manager_pool.queue_time_us":6542,"data dirs.queue_time_us":1094,"data dirs.run_cpu_time_us":201,"data dirs.run_wall_time_us":1291,"drs_written":1,"lbm_read_time_us":64,"lbm_reads_lt_1ms":4,"lbm_write_time_us":584,"lbm_writes_lt_1ms":23,"rows_written":18,"spinlock_wait_cycles":23296,"thread_start_us":7422,"threads_started":2} I0507 06:33:56.813159 15724 maintenance_manager.cc:306] P 41cd5177c3884ec29a257f84325c090b: Scheduling LogGCOp(fdc69131212841d285ea88aadf8b6bd1): free 1840774 bytes of WAL I0507 06:33:56.813596 15618 log_reader.cc:377] T fdc69131212841d285ea88aadf8b6bd1: removed 2 log segments from log reader I0507 06:33:56.813740 15618 log.cc:863] T fdc69131212841d285ea88aadf8b6bd1 P 41cd5177c3884ec29a257f84325c090b: Deleting log segment in path: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_4.1557210500371863-231/raft_consensus-itest-cluster/ts-0/wal/wals/fdc69131212841d285ea88aadf8b6bd1/wal-000000001 (ops 1-9) I0507 06:33:56.813905 15618 log.cc:863] T fdc69131212841d285ea88aadf8b6bd1 P 41cd5177c3884ec29a257f84325c090b: Deleting log segment in path: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_4.1557210500371863-231/raft_consensus-itest-cluster/ts-0/wal/wals/fdc69131212841d285ea88aadf8b6bd1/wal-000000002 (ops 10-16) I0507 06:33:56.814448 15618 maintenance_manager.cc:501] P 41cd5177c3884ec29a257f84325c090b: LogGCOp(fdc69131212841d285ea88aadf8b6bd1) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {} I0507 06:33:56.814749 15724 maintenance_manager.cc:306] P 41cd5177c3884ec29a257f84325c090b: Scheduling FlushMRSOp(fdc69131212841d285ea88aadf8b6bd1): 1840682 bytes log retention I0507 06:33:56.826797 231 raft_consensus-itest-base.cc:277] Waiting for log GC on 1b055b8db383436b94d48378a89755a8 I0507 06:33:56.828353 15969 maintenance_manager.cc:306] P 1b055b8db383436b94d48378a89755a8: Scheduling FlushMRSOp(fdc69131212841d285ea88aadf8b6bd1): 4602072 bytes log retention I0507 06:33:56.842237 15618 maintenance_manager.cc:501] P 41cd5177c3884ec29a257f84325c090b: FlushMRSOp(fdc69131212841d285ea88aadf8b6bd1) complete. Timing: real 0.027s user 0.016s sys 0.004s Metrics: {"bytes_written":135803,"cfile_init":1,"data dirs.queue_time_us":47,"data dirs.run_cpu_time_us":141,"data dirs.run_wall_time_us":478,"drs_written":1,"lbm_read_time_us":48,"lbm_reads_lt_1ms":4,"lbm_write_time_us":575,"lbm_writes_lt_1ms":23,"rows_written":12} I0507 06:33:56.843169 15724 maintenance_manager.cc:306] P 41cd5177c3884ec29a257f84325c090b: Scheduling LogGCOp(fdc69131212841d285ea88aadf8b6bd1): free 920360 bytes of WAL I0507 06:33:56.843400 15618 log_reader.cc:377] T fdc69131212841d285ea88aadf8b6bd1: removed 1 log segments from log reader I0507 06:33:56.843451 15618 log.cc:863] T fdc69131212841d285ea88aadf8b6bd1 P 41cd5177c3884ec29a257f84325c090b: Deleting log segment in path: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_4.1557210500371863-231/raft_consensus-itest-cluster/ts-0/wal/wals/fdc69131212841d285ea88aadf8b6bd1/wal-000000003 (ops 17-23) I0507 06:33:56.843747 15618 maintenance_manager.cc:501] P 41cd5177c3884ec29a257f84325c090b: LogGCOp(fdc69131212841d285ea88aadf8b6bd1) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {} I0507 06:33:56.844135 15724 maintenance_manager.cc:306] P 41cd5177c3884ec29a257f84325c090b: Scheduling UndoDeltaBlockGCOp(fdc69131212841d285ea88aadf8b6bd1): 612 bytes on disk I0507 06:33:56.845255 15618 maintenance_manager.cc:501] P 41cd5177c3884ec29a257f84325c090b: UndoDeltaBlockGCOp(fdc69131212841d285ea88aadf8b6bd1) complete. Timing: real 0.001s user 0.001s sys 0.000s Metrics: {"cfile_init":2,"lbm_read_time_us":128,"lbm_reads_lt_1ms":8} I0507 06:33:56.845655 15724 maintenance_manager.cc:306] P 41cd5177c3884ec29a257f84325c090b: Scheduling CompactRowSetsOp(fdc69131212841d285ea88aadf8b6bd1): perf score=0.915688 I0507 06:33:56.896798 15863 maintenance_manager.cc:501] P 1b055b8db383436b94d48378a89755a8: FlushMRSOp(fdc69131212841d285ea88aadf8b6bd1) complete. Timing: real 0.068s user 0.051s sys 0.013s Metrics: {"bytes_written":136001,"cfile_init":1,"compiler_manager_pool.queue_time_us":10591,"data dirs.queue_time_us":1474,"data dirs.run_cpu_time_us":186,"data dirs.run_wall_time_us":3608,"drs_written":1,"lbm_read_time_us":67,"lbm_reads_lt_1ms":4,"lbm_write_time_us":533,"lbm_writes_lt_1ms":23,"rows_written":36,"thread_start_us":3755,"threads_started":2} I0507 06:33:56.897989 15969 maintenance_manager.cc:306] P 1b055b8db383436b94d48378a89755a8: Scheduling LogGCOp(fdc69131212841d285ea88aadf8b6bd1): free 2761256 bytes of WAL I0507 06:33:56.898298 15863 log_reader.cc:377] T fdc69131212841d285ea88aadf8b6bd1: removed 3 log segments from log reader I0507 06:33:56.898394 15863 log.cc:863] T fdc69131212841d285ea88aadf8b6bd1 P 1b055b8db383436b94d48378a89755a8: Deleting log segment in path: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_4.1557210500371863-231/raft_consensus-itest-cluster/ts-2/wal/wals/fdc69131212841d285ea88aadf8b6bd1/wal-000000001 (ops 1-9) I0507 06:33:56.898502 15863 log.cc:863] T fdc69131212841d285ea88aadf8b6bd1 P 1b055b8db383436b94d48378a89755a8: Deleting log segment in path: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_4.1557210500371863-231/raft_consensus-itest-cluster/ts-2/wal/wals/fdc69131212841d285ea88aadf8b6bd1/wal-000000002 (ops 10-16) I0507 06:33:56.898592 15863 log.cc:863] T fdc69131212841d285ea88aadf8b6bd1 P 1b055b8db383436b94d48378a89755a8: Deleting log segment in path: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_4.1557210500371863-231/raft_consensus-itest-cluster/ts-2/wal/wals/fdc69131212841d285ea88aadf8b6bd1/wal-000000003 (ops 17-23) I0507 06:33:56.899256 15863 maintenance_manager.cc:501] P 1b055b8db383436b94d48378a89755a8: LogGCOp(fdc69131212841d285ea88aadf8b6bd1) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {} I0507 06:33:56.899503 15969 maintenance_manager.cc:306] P 1b055b8db383436b94d48378a89755a8: Scheduling UndoDeltaBlockGCOp(fdc69131212841d285ea88aadf8b6bd1): 536 bytes on disk I0507 06:33:56.900104 15863 maintenance_manager.cc:501] P 1b055b8db383436b94d48378a89755a8: UndoDeltaBlockGCOp(fdc69131212841d285ea88aadf8b6bd1) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":64,"lbm_reads_lt_1ms":4} I0507 06:33:56.908768 231 raft_consensus-itest-base.cc:282] Log GC complete on 1b055b8db383436b94d48378a89755a8 I0507 06:33:56.918081 15618 maintenance_manager.cc:501] P 41cd5177c3884ec29a257f84325c090b: CompactRowSetsOp(fdc69131212841d285ea88aadf8b6bd1) complete. Timing: real 0.072s user 0.051s sys 0.016s Metrics: {"bytes_written":135937,"cfile_cache_hit":10,"cfile_cache_hit_bytes":262392,"cfile_cache_miss":26,"cfile_cache_miss_bytes":264164,"cfile_init":5,"data dirs.queue_time_us":345,"data dirs.run_cpu_time_us":251,"data dirs.run_wall_time_us":631,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":868,"lbm_reads_lt_1ms":46,"lbm_write_time_us":739,"lbm_writes_lt_1ms":23,"num_input_rowsets":2,"rows_written":30} I0507 06:33:56.919088 15724 maintenance_manager.cc:306] P 41cd5177c3884ec29a257f84325c090b: Scheduling UndoDeltaBlockGCOp(fdc69131212841d285ea88aadf8b6bd1): 471 bytes on disk I0507 06:33:56.919692 15618 maintenance_manager.cc:501] P 41cd5177c3884ec29a257f84325c090b: UndoDeltaBlockGCOp(fdc69131212841d285ea88aadf8b6bd1) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":74,"lbm_reads_lt_1ms":4} I0507 06:33:57.060153 16041 consensus_queue.cc:689] T fdc69131212841d285ea88aadf8b6bd1 P 1b055b8db383436b94d48378a89755a8 [LEADER]: The logs necessary to catch up peer 407e162251194eb69b048b59362a51bf have been garbage collected. The follower will never be able to catch up (Not found: Failed to read ops 2..31: Segment 1 which contained index 2 has been GCed) I0507 06:33:57.060449 16041 consensus_queue.cc:544] T fdc69131212841d285ea88aadf8b6bd1 P 1b055b8db383436b94d48378a89755a8 [LEADER]: The logs necessary to catch up peer 407e162251194eb69b048b59362a51bf have been garbage collected. The replica will never be able to catch up I0507 06:33:57.063984 15557 catalog_manager.cc:3739] Sending ChangeConfig:REMOVE_PEER on tablet fdc69131212841d285ea88aadf8b6bd1 (attempt 1) I0507 06:33:57.066299 15925 consensus_queue.cc:206] T fdc69131212841d285ea88aadf8b6bd1 P 1b055b8db383436b94d48378a89755a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 38, Committed index: 38, Last appended: 2.38, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 39 OBSOLETE_local: false peers { permanent_uuid: "1b055b8db383436b94d48378a89755a8" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 37465 } } peers { permanent_uuid: "41cd5177c3884ec29a257f84325c090b" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 33409 } } I0507 06:33:57.068423 15679 raft_consensus.cc:1181] T fdc69131212841d285ea88aadf8b6bd1 P 41cd5177c3884ec29a257f84325c090b [term 2 FOLLOWER]: Refusing update from remote peer 1b055b8db383436b94d48378a89755a8: Log matching property violated. Preceding OpId in replica: term: 2 index: 38. Preceding OpId from leader: term: 2 index: 39. (index mismatch) I0507 06:33:57.069048 16041 consensus_queue.cc:984] T fdc69131212841d285ea88aadf8b6bd1 P 1b055b8db383436b94d48378a89755a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "41cd5177c3884ec29a257f84325c090b" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 33409 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 39, Last known committed idx: 38, Time since last communication: 0.000s I0507 06:33:57.071990 16045 raft_consensus.cc:2788] T fdc69131212841d285ea88aadf8b6bd1 P 1b055b8db383436b94d48378a89755a8 [term 2 LEADER]: Committing config change with OpId 2.39: config changed from index -1 to 39, VOTER 407e162251194eb69b048b59362a51bf (127.0.57.194) evicted. New config: { opid_index: 39 OBSOLETE_local: false peers { permanent_uuid: "1b055b8db383436b94d48378a89755a8" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 37465 } } peers { permanent_uuid: "41cd5177c3884ec29a257f84325c090b" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 33409 } } } I0507 06:33:57.072477 15679 raft_consensus.cc:2788] T fdc69131212841d285ea88aadf8b6bd1 P 41cd5177c3884ec29a257f84325c090b [term 2 FOLLOWER]: Committing config change with OpId 2.39: config changed from index -1 to 39, VOTER 407e162251194eb69b048b59362a51bf (127.0.57.194) evicted. New config: { opid_index: 39 OBSOLETE_local: false peers { permanent_uuid: "1b055b8db383436b94d48378a89755a8" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 37465 } } peers { permanent_uuid: "41cd5177c3884ec29a257f84325c090b" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 33409 } } } I0507 06:33:57.075907 15542 catalog_manager.cc:3531] ChangeConfig:REMOVE_PEER RPC for tablet fdc69131212841d285ea88aadf8b6bd1 with cas_config_opid_index -1: ChangeConfig:REMOVE_PEER succeeded (attempt 1) I0507 06:33:57.076862 15557 catalog_manager.cc:3988] T fdc69131212841d285ea88aadf8b6bd1 P 1b055b8db383436b94d48378a89755a8 reported cstate change: config changed from index -1 to 39, VOTER 407e162251194eb69b048b59362a51bf (127.0.57.194) evicted. New cstate: current_term: 2 leader_uuid: "1b055b8db383436b94d48378a89755a8" committed_config { opid_index: 39 OBSOLETE_local: false peers { permanent_uuid: "1b055b8db383436b94d48378a89755a8" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 37465 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "41cd5177c3884ec29a257f84325c090b" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 33409 } health_report { overall_health: HEALTHY } } } I0507 06:33:57.089684 15557 catalog_manager.cc:3390] Sending DeleteTablet(TABLET_DATA_TOMBSTONED) for tablet fdc69131212841d285ea88aadf8b6bd1 on 407e162251194eb69b048b59362a51bf (127.0.57.194:32837) (TS 407e162251194eb69b048b59362a51bf not found in new config with opid_index 39) I0507 06:33:57.092010 15557 catalog_manager.cc:3616] Sending ChangeConfig:ADD_PEER:NON_VOTER on tablet fdc69131212841d285ea88aadf8b6bd1 (attempt 1) W0507 06:33:57.093222 15544 catalog_manager.cc:3132] TS 407e162251194eb69b048b59362a51bf (127.0.57.194:32837): Delete Tablet RPC failed for tablet fdc69131212841d285ea88aadf8b6bd1: Network error: Client connection negotiation failed: client connection to 127.0.57.194:32837: connect: Connection refused (error 111) I0507 06:33:57.093323 15544 catalog_manager.cc:3168] Scheduling retry of fdc69131212841d285ea88aadf8b6bd1 Delete Tablet RPC for TS=407e162251194eb69b048b59362a51bf with a delay of 32 ms (attempt = 1) I0507 06:33:57.093945 15925 consensus_queue.cc:206] T fdc69131212841d285ea88aadf8b6bd1 P 1b055b8db383436b94d48378a89755a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 39, Committed index: 39, Last appended: 2.39, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 40 OBSOLETE_local: false peers { permanent_uuid: "1b055b8db383436b94d48378a89755a8" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 37465 } } peers { permanent_uuid: "41cd5177c3884ec29a257f84325c090b" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 33409 } } peers { permanent_uuid: "407e162251194eb69b048b59362a51bf" member_type: NON_VOTER last_known_addr { host: "127.0.57.194" port: 32837 } attrs { promote: true } } W0507 06:33:57.095978 15859 consensus_peers.cc:457] T fdc69131212841d285ea88aadf8b6bd1 P 1b055b8db383436b94d48378a89755a8 -> Peer 407e162251194eb69b048b59362a51bf (127.0.57.194:32837): Couldn't send request to peer 407e162251194eb69b048b59362a51bf. Status: Network error: Client connection negotiation failed: client connection to 127.0.57.194:32837: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry. I0507 06:33:57.096282 15679 raft_consensus.cc:1181] T fdc69131212841d285ea88aadf8b6bd1 P 41cd5177c3884ec29a257f84325c090b [term 2 FOLLOWER]: Refusing update from remote peer 1b055b8db383436b94d48378a89755a8: Log matching property violated. Preceding OpId in replica: term: 2 index: 39. Preceding OpId from leader: term: 2 index: 40. (index mismatch) I0507 06:33:57.096786 16008 consensus_queue.cc:984] T fdc69131212841d285ea88aadf8b6bd1 P 1b055b8db383436b94d48378a89755a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "41cd5177c3884ec29a257f84325c090b" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 33409 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 40, Last known committed idx: 39, Time since last communication: 0.000s I0507 06:33:57.098989 16041 raft_consensus.cc:2788] T fdc69131212841d285ea88aadf8b6bd1 P 1b055b8db383436b94d48378a89755a8 [term 2 LEADER]: Committing config change with OpId 2.40: config changed from index 39 to 40, NON_VOTER 407e162251194eb69b048b59362a51bf (127.0.57.194) added. New config: { opid_index: 40 OBSOLETE_local: false peers { permanent_uuid: "1b055b8db383436b94d48378a89755a8" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 37465 } } peers { permanent_uuid: "41cd5177c3884ec29a257f84325c090b" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 33409 } } peers { permanent_uuid: "407e162251194eb69b048b59362a51bf" member_type: NON_VOTER last_known_addr { host: "127.0.57.194" port: 32837 } attrs { promote: true } } } I0507 06:33:57.099577 15679 raft_consensus.cc:2788] T fdc69131212841d285ea88aadf8b6bd1 P 41cd5177c3884ec29a257f84325c090b [term 2 FOLLOWER]: Committing config change with OpId 2.40: config changed from index 39 to 40, NON_VOTER 407e162251194eb69b048b59362a51bf (127.0.57.194) added. New config: { opid_index: 40 OBSOLETE_local: false peers { permanent_uuid: "1b055b8db383436b94d48378a89755a8" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 37465 } } peers { permanent_uuid: "41cd5177c3884ec29a257f84325c090b" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 33409 } } peers { permanent_uuid: "407e162251194eb69b048b59362a51bf" member_type: NON_VOTER last_known_addr { host: "127.0.57.194" port: 32837 } attrs { promote: true } } } I0507 06:33:57.102236 15542 catalog_manager.cc:3531] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet fdc69131212841d285ea88aadf8b6bd1 with cas_config_opid_index 39: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1) I0507 06:33:57.103272 15557 catalog_manager.cc:3988] T fdc69131212841d285ea88aadf8b6bd1 P 1b055b8db383436b94d48378a89755a8 reported cstate change: config changed from index 39 to 40, NON_VOTER 407e162251194eb69b048b59362a51bf (127.0.57.194) added. New cstate: current_term: 2 leader_uuid: "1b055b8db383436b94d48378a89755a8" committed_config { opid_index: 40 OBSOLETE_local: false peers { permanent_uuid: "1b055b8db383436b94d48378a89755a8" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 37465 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "41cd5177c3884ec29a257f84325c090b" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 33409 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "407e162251194eb69b048b59362a51bf" member_type: NON_VOTER last_known_addr { host: "127.0.57.194" port: 32837 } attrs { promote: true } health_report { overall_health: UNKNOWN } } } I0507 06:33:57.125917 15544 catalog_manager.cc:3390] Sending DeleteTablet(TABLET_DATA_TOMBSTONED) for tablet fdc69131212841d285ea88aadf8b6bd1 on 407e162251194eb69b048b59362a51bf (127.0.57.194:32837) (TS 407e162251194eb69b048b59362a51bf not found in new config with opid_index 39) W0507 06:33:57.126895 15544 catalog_manager.cc:3132] TS 407e162251194eb69b048b59362a51bf (127.0.57.194:32837): Delete Tablet RPC failed for tablet fdc69131212841d285ea88aadf8b6bd1: Network error: Client connection negotiation failed: client connection to 127.0.57.194:32837: connect: Connection refused (error 111) I0507 06:33:57.126987 15544 catalog_manager.cc:3168] Scheduling retry of fdc69131212841d285ea88aadf8b6bd1 Delete Tablet RPC for TS=407e162251194eb69b048b59362a51bf with a delay of 79 ms (attempt = 2) I0507 06:33:57.206759 15544 catalog_manager.cc:3390] Sending DeleteTablet(TABLET_DATA_TOMBSTONED) for tablet fdc69131212841d285ea88aadf8b6bd1 on 407e162251194eb69b048b59362a51bf (127.0.57.194:32837) (TS 407e162251194eb69b048b59362a51bf not found in new config with opid_index 39) W0507 06:33:57.207693 15544 catalog_manager.cc:3132] TS 407e162251194eb69b048b59362a51bf (127.0.57.194:32837): Delete Tablet RPC failed for tablet fdc69131212841d285ea88aadf8b6bd1: Network error: Client connection negotiation failed: client connection to 127.0.57.194:32837: connect: Connection refused (error 111) I0507 06:33:57.207773 15544 catalog_manager.cc:3168] Scheduling retry of fdc69131212841d285ea88aadf8b6bd1 Delete Tablet RPC for TS=407e162251194eb69b048b59362a51bf with a delay of 112 ms (attempt = 3) I0507 06:33:57.320869 15544 catalog_manager.cc:3390] Sending DeleteTablet(TABLET_DATA_TOMBSTONED) for tablet fdc69131212841d285ea88aadf8b6bd1 on 407e162251194eb69b048b59362a51bf (127.0.57.194:32837) (TS 407e162251194eb69b048b59362a51bf not found in new config with opid_index 39) W0507 06:33:57.322031 15544 catalog_manager.cc:3132] TS 407e162251194eb69b048b59362a51bf (127.0.57.194:32837): Delete Tablet RPC failed for tablet fdc69131212841d285ea88aadf8b6bd1: Network error: Client connection negotiation failed: client connection to 127.0.57.194:32837: connect: Connection refused (error 111) I0507 06:33:57.322127 15544 catalog_manager.cc:3168] Scheduling retry of fdc69131212841d285ea88aadf8b6bd1 Delete Tablet RPC for TS=407e162251194eb69b048b59362a51bf with a delay of 175 ms (attempt = 4) I0507 06:33:57.498018 15544 catalog_manager.cc:3390] Sending DeleteTablet(TABLET_DATA_TOMBSTONED) for tablet fdc69131212841d285ea88aadf8b6bd1 on 407e162251194eb69b048b59362a51bf (127.0.57.194:32837) (TS 407e162251194eb69b048b59362a51bf not found in new config with opid_index 39) W0507 06:33:57.499164 15544 catalog_manager.cc:3132] TS 407e162251194eb69b048b59362a51bf (127.0.57.194:32837): Delete Tablet RPC failed for tablet fdc69131212841d285ea88aadf8b6bd1: Network error: Client connection negotiation failed: client connection to 127.0.57.194:32837: connect: Connection refused (error 111) I0507 06:33:57.499244 15544 catalog_manager.cc:3168] Scheduling retry of fdc69131212841d285ea88aadf8b6bd1 Delete Tablet RPC for TS=407e162251194eb69b048b59362a51bf with a delay of 278 ms (attempt = 5) I0507 06:33:57.778518 15544 catalog_manager.cc:3390] Sending DeleteTablet(TABLET_DATA_TOMBSTONED) for tablet fdc69131212841d285ea88aadf8b6bd1 on 407e162251194eb69b048b59362a51bf (127.0.57.194:32837) (TS 407e162251194eb69b048b59362a51bf not found in new config with opid_index 39) W0507 06:33:57.779635 15544 catalog_manager.cc:3132] TS 407e162251194eb69b048b59362a51bf (127.0.57.194:32837): Delete Tablet RPC failed for tablet fdc69131212841d285ea88aadf8b6bd1: Network error: Client connection negotiation failed: client connection to 127.0.57.194:32837: connect: Connection refused (error 111) I0507 06:33:57.779732 15544 catalog_manager.cc:3168] Scheduling retry of fdc69131212841d285ea88aadf8b6bd1 Delete Tablet RPC for TS=407e162251194eb69b048b59362a51bf with a delay of 528 ms (attempt = 6) I0507 06:33:58.308590 15544 catalog_manager.cc:3390] Sending DeleteTablet(TABLET_DATA_TOMBSTONED) for tablet fdc69131212841d285ea88aadf8b6bd1 on 407e162251194eb69b048b59362a51bf (127.0.57.194:32837) (TS 407e162251194eb69b048b59362a51bf not found in new config with opid_index 39) W0507 06:33:58.310132 15544 catalog_manager.cc:3132] TS 407e162251194eb69b048b59362a51bf (127.0.57.194:32837): Delete Tablet RPC failed for tablet fdc69131212841d285ea88aadf8b6bd1: Network error: Client connection negotiation failed: client connection to 127.0.57.194:32837: connect: Connection refused (error 111) I0507 06:33:58.310215 15544 catalog_manager.cc:3168] Scheduling retry of fdc69131212841d285ea88aadf8b6bd1 Delete Tablet RPC for TS=407e162251194eb69b048b59362a51bf with a delay of 1060 ms (attempt = 7) I0507 06:33:58.910619 231 raft_consensus-itest-base.cc:298] Servers converged with original term 2 I0507 06:33:58.918859 231 external_mini_cluster.cc:933] Running /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_4.1557210500371863-231/raft_consensus-itest-cluster/ts-1/wal --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_4.1557210500371863-231/raft_consensus-itest-cluster/ts-1/data --block_manager=log --rpc_bind_addresses=127.0.57.194:32837 --local_ip_for_outbound_sockets=127.0.57.194 --webserver_port=44291 --webserver_interface=127.0.57.194 --tserver_master_addrs=127.0.57.254:34189 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_4.1557210500371863-231/raft_consensus-itest-cluster/ts-1/logs --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_4.1557210500371863-231/raft_consensus-itest-cluster/ts-1/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=true --follower_unavailable_considered_failed_sec=5 --log_compression_codec=none --log_cache_size_limit_mb=1 --log_segment_size_mb=1 --log_async_preallocate_segments=false --log_min_segments_to_retain=1 --log_max_segments_to_retain=3 --maintenance_manager_polling_interval_ms=100 --log_target_replay_size_mb=1 --max_cell_size_bytes=1000000 with env {} WARNING: Logging before InitGoogleLogging() is written to STDERR W0507 06:33:59.088511 16056 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0507 06:33:59.088863 16056 flags.cc:404] Enabled unsafe flag: --max_cell_size_bytes=1000000 W0507 06:33:59.088907 16056 flags.cc:404] Enabled unsafe flag: --never_fsync=true W0507 06:33:59.090962 16056 flags.cc:404] Enabled experimental flag: --log_compression_codec=none W0507 06:33:59.091027 16056 flags.cc:404] Enabled experimental flag: --log_max_segments_to_retain=3 W0507 06:33:59.091060 16056 flags.cc:404] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true W0507 06:33:59.091110 16056 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W0507 06:33:59.091200 16056 flags.cc:404] Enabled experimental flag: --log_target_replay_size_mb=1 W0507 06:33:59.091249 16056 flags.cc:404] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.57.194 I0507 06:33:59.095728 16056 tablet_server_main.cc:78] Tablet server non-default flags: --follower_unavailable_considered_failed_sec=5 --log_compression_codec=none --log_max_segments_to_retain=3 --log_cache_size_limit_mb=1 --log_async_preallocate_segments=false --log_segment_size_mb=1 --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_4.1557210500371863-231/raft_consensus-itest-cluster/ts-1/data --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_4.1557210500371863-231/raft_consensus-itest-cluster/ts-1/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.0.57.194:32837 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_4.1557210500371863-231/raft_consensus-itest-cluster/ts-1/data/info.pb --webserver_interface=127.0.57.194 --webserver_port=44291 --max_cell_size_bytes=1000000 --tserver_master_addrs=127.0.57.254:34189 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --log_target_replay_size_mb=1 --maintenance_manager_polling_interval_ms=100 --enable_minidumps=false --local_ip_for_outbound_sockets=127.0.57.194 --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_4.1557210500371863-231/raft_consensus-itest-cluster/ts-1/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.10.0-cdh6.x-SNAPSHOT revision 7d0ee26697926e87af9ef395008799577f6ab43f build type FASTDEBUG built by jenkins at 06 May 2019 22:44:21 PST on quasar-knmpfx-1.vpc.cloudera.com build id 322017 ASAN enabled I0507 06:33:59.104331 16056 system_ntp.cc:202] NTP initialized. Skew: 500ppm Current error: 194016us I0507 06:33:59.105092 16056 fs_manager.cc:263] Metadata directory not provided I0507 06:33:59.105212 16056 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0507 06:33:59.112264 16056 fs_manager.cc:399] Time spent opening directory manager: real 0.005s user 0.006s sys 0.000s I0507 06:33:59.113075 16056 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0507 06:33:59.113719 16056 file_cache.cc:466] Constructed file cache lbm with capacity 419430 I0507 06:33:59.116855 16056 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s I0507 06:33:59.116947 16056 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_4.1557210500371863-231/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_4.1557210500371863-231/raft_consensus-itest-cluster/ts-1/wal uuid: "407e162251194eb69b048b59362a51bf" format_stamp: "Formatted at 2019-05-07 06:33:53 on dist-test-slave-dist-test-slave-rqp6" I0507 06:33:59.117949 16056 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_4.1557210500371863-231/raft_consensus-itest-cluster/ts-1/wal metadata directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_4.1557210500371863-231/raft_consensus-itest-cluster/ts-1/wal 1 data directories: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_4.1557210500371863-231/raft_consensus-itest-cluster/ts-1/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0507 06:33:59.150408 16056 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0507 06:33:59.152360 16056 ts_tablet_manager.cc:344] Loading tablet metadata (0/1 complete) I0507 06:33:59.156472 16056 ts_tablet_manager.cc:356] Loaded tablet metadata (1 total tablets, 1 live tablets) I0507 06:33:59.156559 16056 ts_tablet_manager.cc:362] Registering tablets (0/1 complete) I0507 06:33:59.159708 16071 tablet_bootstrap.cc:439] T fdc69131212841d285ea88aadf8b6bd1 P 407e162251194eb69b048b59362a51bf: Bootstrap starting. I0507 06:33:59.167654 16071 log.cc:525] T fdc69131212841d285ea88aadf8b6bd1 P 407e162251194eb69b048b59362a51bf: Log is configured to *not* fsync() on all Append() calls I0507 06:33:59.168066 16056 ts_tablet_manager.cc:376] Registered 1 tablets I0507 06:33:59.171173 16071 tablet_bootstrap.cc:439] T fdc69131212841d285ea88aadf8b6bd1 P 407e162251194eb69b048b59362a51bf: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I0507 06:33:59.171793 16071 tablet_bootstrap.cc:439] T fdc69131212841d285ea88aadf8b6bd1 P 407e162251194eb69b048b59362a51bf: Bootstrap complete. I0507 06:33:59.172168 16071 ts_tablet_manager.cc:1053] T fdc69131212841d285ea88aadf8b6bd1 P 407e162251194eb69b048b59362a51bf: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.006s I0507 06:33:59.182503 16071 raft_consensus.cc:340] T fdc69131212841d285ea88aadf8b6bd1 P 407e162251194eb69b048b59362a51bf [term 1 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b055b8db383436b94d48378a89755a8" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 37465 } } peers { permanent_uuid: "41cd5177c3884ec29a257f84325c090b" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 33409 } } peers { permanent_uuid: "407e162251194eb69b048b59362a51bf" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 32837 } } I0507 06:33:59.185633 16071 raft_consensus.cc:705] T fdc69131212841d285ea88aadf8b6bd1 P 407e162251194eb69b048b59362a51bf [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 407e162251194eb69b048b59362a51bf, State: Initialized, Role: FOLLOWER I0507 06:33:59.186342 16071 consensus_queue.cc:229] T fdc69131212841d285ea88aadf8b6bd1 P 407e162251194eb69b048b59362a51bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b055b8db383436b94d48378a89755a8" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 37465 } } peers { permanent_uuid: "41cd5177c3884ec29a257f84325c090b" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 33409 } } peers { permanent_uuid: "407e162251194eb69b048b59362a51bf" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 32837 } } I0507 06:33:59.187916 16071 ts_tablet_manager.cc:1081] T fdc69131212841d285ea88aadf8b6bd1 P 407e162251194eb69b048b59362a51bf: Time spent starting tablet: real 0.016s user 0.011s sys 0.003s I0507 06:33:59.212244 16056 rpc_server.cc:205] RPC server started. Bound to: 127.0.57.194:32837 I0507 06:33:59.213474 16056 webserver.cc:308] Webserver started at http://127.0.57.194:44291/ using document root and password file I0507 06:33:59.214151 16056 server_base.cc:608] Dumped server information to /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_4.1557210500371863-231/raft_consensus-itest-cluster/ts-1/data/info.pb I0507 06:33:59.216737 231 external_mini_cluster.cc:995] Started /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver as pid 16056 I0507 06:33:59.223783 16178 heartbeater.cc:345] Connected to a master server at 127.0.57.254:34189 I0507 06:33:59.223970 16178 heartbeater.cc:426] Registering TS with master... I0507 06:33:59.224597 16178 heartbeater.cc:475] Master 127.0.57.254:34189 requested a full tablet report, sending... I0507 06:33:59.226301 15560 ts_manager.cc:159] Re-registered known tserver with Master: 407e162251194eb69b048b59362a51bf (127.0.57.194:32837) I0507 06:33:59.227413 15560 master_service.cc:273] Signed X509 certificate for tserver {username='slave'} at 127.0.57.194:43907 I0507 06:33:59.371496 15544 catalog_manager.cc:3390] Sending DeleteTablet(TABLET_DATA_TOMBSTONED) for tablet fdc69131212841d285ea88aadf8b6bd1 on 407e162251194eb69b048b59362a51bf (127.0.57.194:32837) (TS 407e162251194eb69b048b59362a51bf not found in new config with opid_index 39) I0507 06:33:59.378365 16115 tablet_service.cc:1057] Processing DeleteTablet for tablet fdc69131212841d285ea88aadf8b6bd1 with delete_type TABLET_DATA_TOMBSTONED (TS 407e162251194eb69b048b59362a51bf not found in new config with opid_index 39) from {username='slave'} at 127.0.0.1:58800 I0507 06:33:59.380125 16184 tablet_replica.cc:264] T fdc69131212841d285ea88aadf8b6bd1 P 407e162251194eb69b048b59362a51bf: stopping tablet replica I0507 06:33:59.380455 16184 raft_consensus.cc:2143] T fdc69131212841d285ea88aadf8b6bd1 P 407e162251194eb69b048b59362a51bf [term 1 FOLLOWER]: Raft consensus shutting down. I0507 06:33:59.380761 16184 raft_consensus.cc:2170] T fdc69131212841d285ea88aadf8b6bd1 P 407e162251194eb69b048b59362a51bf [term 1 FOLLOWER]: Raft consensus is shut down! I0507 06:33:59.382093 16184 ts_tablet_manager.cc:1377] T fdc69131212841d285ea88aadf8b6bd1 P 407e162251194eb69b048b59362a51bf: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED I0507 06:33:59.387101 16184 ts_tablet_manager.cc:1390] T fdc69131212841d285ea88aadf8b6bd1 P 407e162251194eb69b048b59362a51bf: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.1 I0507 06:33:59.387204 16184 log.cc:990] T fdc69131212841d285ea88aadf8b6bd1 P 407e162251194eb69b048b59362a51bf: Deleting WAL directory at /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_4.1557210500371863-231/raft_consensus-itest-cluster/ts-1/wal/wals/fdc69131212841d285ea88aadf8b6bd1 I0507 06:33:59.388259 15544 catalog_manager.cc:3368] TS 407e162251194eb69b048b59362a51bf (127.0.57.194:32837): tablet fdc69131212841d285ea88aadf8b6bd1 (table TestTable [id=a9924983367f4c9ea12957ce08a021a0]) successfully deleted W0507 06:33:59.392581 15859 consensus_peers.cc:457] T fdc69131212841d285ea88aadf8b6bd1 P 1b055b8db383436b94d48378a89755a8 -> Peer 407e162251194eb69b048b59362a51bf (127.0.57.194:32837): Couldn't send request to peer 407e162251194eb69b048b59362a51bf. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 6: this message will repeat every 5th retry. I0507 06:33:59.984145 16185 ts_tablet_manager.cc:676] T fdc69131212841d285ea88aadf8b6bd1 P 407e162251194eb69b048b59362a51bf: Initiating tablet copy from peer 1b055b8db383436b94d48378a89755a8 (127.0.57.195:37465) I0507 06:33:59.984606 16185 tablet_copy_client.cc:240] T fdc69131212841d285ea88aadf8b6bd1 P 407e162251194eb69b048b59362a51bf: tablet copy: Beginning tablet copy session from remote peer at address 127.0.57.195:37465 I0507 06:33:59.990288 15945 tablet_copy_service.cc:135] P 1b055b8db383436b94d48378a89755a8: Received BeginTabletCopySession request for tablet fdc69131212841d285ea88aadf8b6bd1 from peer 407e162251194eb69b048b59362a51bf ({username='slave'} at 127.0.57.194:39209) I0507 06:33:59.990485 15945 tablet_copy_service.cc:156] P 1b055b8db383436b94d48378a89755a8: Beginning new tablet copy session on tablet fdc69131212841d285ea88aadf8b6bd1 from peer 407e162251194eb69b048b59362a51bf at {username='slave'} at 127.0.57.194:39209: session id = 407e162251194eb69b048b59362a51bf-fdc69131212841d285ea88aadf8b6bd1 I0507 06:33:59.993141 15945 tablet_copy_source_session.cc:202] T fdc69131212841d285ea88aadf8b6bd1 P 1b055b8db383436b94d48378a89755a8: Tablet Copy: opened 5 blocks and 3 log segments I0507 06:33:59.995076 16185 ts_tablet_manager.cc:1377] T fdc69131212841d285ea88aadf8b6bd1 P 407e162251194eb69b048b59362a51bf: Deleting tablet data with delete state TABLET_DATA_COPYING I0507 06:33:59.998806 16185 ts_tablet_manager.cc:1390] T fdc69131212841d285ea88aadf8b6bd1 P 407e162251194eb69b048b59362a51bf: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.1 I0507 06:33:59.999138 16185 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet fdc69131212841d285ea88aadf8b6bd1. 1 dirs total, 0 dirs full, 0 dirs failed I0507 06:34:00.001358 16185 tablet_copy_client.cc:575] T fdc69131212841d285ea88aadf8b6bd1 P 407e162251194eb69b048b59362a51bf: tablet copy: Starting download of 5 data blocks... I0507 06:34:00.010557 16185 tablet_copy_client.cc:538] T fdc69131212841d285ea88aadf8b6bd1 P 407e162251194eb69b048b59362a51bf: tablet copy: Starting download of 3 WAL segments... I0507 06:34:00.024430 16185 tablet_copy_client.cc:414] T fdc69131212841d285ea88aadf8b6bd1 P 407e162251194eb69b048b59362a51bf: tablet copy: Tablet Copy complete. Replacing tablet superblock. I0507 06:34:00.027362 16185 tablet_bootstrap.cc:439] T fdc69131212841d285ea88aadf8b6bd1 P 407e162251194eb69b048b59362a51bf: Bootstrap starting. I0507 06:34:00.043202 16185 tablet_bootstrap.cc:439] T fdc69131212841d285ea88aadf8b6bd1 P 407e162251194eb69b048b59362a51bf: Bootstrap replayed 1/3 log segments. Stats: ops{read=7 overwritten=0 applied=5 ignored=5} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=2. Pending: 2 replicates I0507 06:34:00.050029 16185 tablet_bootstrap.cc:439] T fdc69131212841d285ea88aadf8b6bd1 P 407e162251194eb69b048b59362a51bf: Bootstrap replayed 2/3 log segments. Stats: ops{read=14 overwritten=0 applied=12 ignored=12} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=2. Pending: 2 replicates I0507 06:34:00.052851 16185 tablet_bootstrap.cc:439] T fdc69131212841d285ea88aadf8b6bd1 P 407e162251194eb69b048b59362a51bf: Bootstrap replayed 3/3 log segments. Stats: ops{read=17 overwritten=0 applied=17 ignored=15} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=2. Pending: 0 replicates I0507 06:34:00.053202 16185 tablet_bootstrap.cc:439] T fdc69131212841d285ea88aadf8b6bd1 P 407e162251194eb69b048b59362a51bf: Bootstrap complete. I0507 06:34:00.053735 16185 ts_tablet_manager.cc:1053] T fdc69131212841d285ea88aadf8b6bd1 P 407e162251194eb69b048b59362a51bf: Time spent bootstrapping tablet: real 0.026s user 0.018s sys 0.009s I0507 06:34:00.054512 16185 raft_consensus.cc:340] T fdc69131212841d285ea88aadf8b6bd1 P 407e162251194eb69b048b59362a51bf [term 2 LEARNER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: 40 OBSOLETE_local: false peers { permanent_uuid: "1b055b8db383436b94d48378a89755a8" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 37465 } } peers { permanent_uuid: "41cd5177c3884ec29a257f84325c090b" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 33409 } } peers { permanent_uuid: "407e162251194eb69b048b59362a51bf" member_type: NON_VOTER last_known_addr { host: "127.0.57.194" port: 32837 } attrs { promote: true } } I0507 06:34:00.054700 16185 raft_consensus.cc:705] T fdc69131212841d285ea88aadf8b6bd1 P 407e162251194eb69b048b59362a51bf [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 407e162251194eb69b048b59362a51bf, State: Initialized, Role: LEARNER I0507 06:34:00.054854 16185 consensus_queue.cc:229] T fdc69131212841d285ea88aadf8b6bd1 P 407e162251194eb69b048b59362a51bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 40, Last appended: 2.40, Last appended by leader: 40, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 40 OBSOLETE_local: false peers { permanent_uuid: "1b055b8db383436b94d48378a89755a8" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 37465 } } peers { permanent_uuid: "41cd5177c3884ec29a257f84325c090b" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 33409 } } peers { permanent_uuid: "407e162251194eb69b048b59362a51bf" member_type: NON_VOTER last_known_addr { host: "127.0.57.194" port: 32837 } attrs { promote: true } } I0507 06:34:00.055490 16185 ts_tablet_manager.cc:1081] T fdc69131212841d285ea88aadf8b6bd1 P 407e162251194eb69b048b59362a51bf: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s I0507 06:34:00.055533 16178 heartbeater.cc:467] Master 127.0.57.254:34189 was elected leader, sending a full tablet report... I0507 06:34:00.056373 15945 tablet_copy_service.cc:325] P 1b055b8db383436b94d48378a89755a8: Request end of tablet copy session 407e162251194eb69b048b59362a51bf-fdc69131212841d285ea88aadf8b6bd1 received from {username='slave'} at 127.0.57.194:39209 I0507 06:34:00.056509 15945 tablet_copy_service.cc:413] P 1b055b8db383436b94d48378a89755a8: ending tablet copy session 407e162251194eb69b048b59362a51bf-fdc69131212841d285ea88aadf8b6bd1 on tablet fdc69131212841d285ea88aadf8b6bd1 with peer 407e162251194eb69b048b59362a51bf I0507 06:34:00.119254 16179 maintenance_manager.cc:306] P 407e162251194eb69b048b59362a51bf: Scheduling UndoDeltaBlockGCOp(fdc69131212841d285ea88aadf8b6bd1): 536 bytes on disk I0507 06:34:00.120987 16070 maintenance_manager.cc:501] P 407e162251194eb69b048b59362a51bf: UndoDeltaBlockGCOp(fdc69131212841d285ea88aadf8b6bd1) complete. Timing: real 0.001s user 0.001s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":106,"lbm_reads_lt_1ms":4} I0507 06:34:00.448724 16135 raft_consensus.cc:1126] T fdc69131212841d285ea88aadf8b6bd1 P 407e162251194eb69b048b59362a51bf [term 2 LEARNER]: Deduplicated request from leader. Original: 2.39->[2.40-2.40] Dedup: 2.40->[] I0507 06:34:00.514590 15969 maintenance_manager.cc:306] P 1b055b8db383436b94d48378a89755a8: Scheduling LogGCOp(fdc69131212841d285ea88aadf8b6bd1): free 1840816 bytes of WAL I0507 06:34:00.514961 15863 log_reader.cc:377] T fdc69131212841d285ea88aadf8b6bd1: removed 2 log segments from log reader I0507 06:34:00.515033 15863 log.cc:863] T fdc69131212841d285ea88aadf8b6bd1 P 1b055b8db383436b94d48378a89755a8: Deleting log segment in path: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_4.1557210500371863-231/raft_consensus-itest-cluster/ts-2/wal/wals/fdc69131212841d285ea88aadf8b6bd1/wal-000000004 (ops 24-30) I0507 06:34:00.515113 15863 log.cc:863] T fdc69131212841d285ea88aadf8b6bd1 P 1b055b8db383436b94d48378a89755a8: Deleting log segment in path: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_4.1557210500371863-231/raft_consensus-itest-cluster/ts-2/wal/wals/fdc69131212841d285ea88aadf8b6bd1/wal-000000005 (ops 31-37) I0507 06:34:00.515586 15863 maintenance_manager.cc:501] P 1b055b8db383436b94d48378a89755a8: LogGCOp(fdc69131212841d285ea88aadf8b6bd1) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {} I0507 06:34:00.836942 15724 maintenance_manager.cc:306] P 41cd5177c3884ec29a257f84325c090b: Scheduling LogGCOp(fdc69131212841d285ea88aadf8b6bd1): free 920322 bytes of WAL I0507 06:34:00.837343 15618 log_reader.cc:377] T fdc69131212841d285ea88aadf8b6bd1: removed 1 log segments from log reader I0507 06:34:00.837397 15618 log.cc:863] T fdc69131212841d285ea88aadf8b6bd1 P 41cd5177c3884ec29a257f84325c090b: Deleting log segment in path: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_4.1557210500371863-231/raft_consensus-itest-cluster/ts-0/wal/wals/fdc69131212841d285ea88aadf8b6bd1/wal-000000004 (ops 24-30) I0507 06:34:00.837740 15618 maintenance_manager.cc:501] P 41cd5177c3884ec29a257f84325c090b: LogGCOp(fdc69131212841d285ea88aadf8b6bd1) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {} I0507 06:34:00.999090 16191 raft_consensus.cc:974] T fdc69131212841d285ea88aadf8b6bd1 P 1b055b8db383436b94d48378a89755a8: attempting to promote NON_VOTER 407e162251194eb69b048b59362a51bf to VOTER I0507 06:34:00.999747 16191 consensus_queue.cc:206] T fdc69131212841d285ea88aadf8b6bd1 P 1b055b8db383436b94d48378a89755a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 40, Committed index: 40, Last appended: 2.40, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 41 OBSOLETE_local: false peers { permanent_uuid: "1b055b8db383436b94d48378a89755a8" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 37465 } } peers { permanent_uuid: "41cd5177c3884ec29a257f84325c090b" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 33409 } } peers { permanent_uuid: "407e162251194eb69b048b59362a51bf" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 32837 } attrs { promote: false } } I0507 06:34:01.002349 16135 raft_consensus.cc:1181] T fdc69131212841d285ea88aadf8b6bd1 P 407e162251194eb69b048b59362a51bf [term 2 LEARNER]: Refusing update from remote peer 1b055b8db383436b94d48378a89755a8: Log matching property violated. Preceding OpId in replica: term: 2 index: 40. Preceding OpId from leader: term: 2 index: 41. (index mismatch) I0507 06:34:01.002501 15679 raft_consensus.cc:1181] T fdc69131212841d285ea88aadf8b6bd1 P 41cd5177c3884ec29a257f84325c090b [term 2 FOLLOWER]: Refusing update from remote peer 1b055b8db383436b94d48378a89755a8: Log matching property violated. Preceding OpId in replica: term: 2 index: 40. Preceding OpId from leader: term: 2 index: 41. (index mismatch) I0507 06:34:01.003010 16192 consensus_queue.cc:984] T fdc69131212841d285ea88aadf8b6bd1 P 1b055b8db383436b94d48378a89755a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "407e162251194eb69b048b59362a51bf" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 32837 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 41, Last known committed idx: 40, Time since last communication: 0.000s I0507 06:34:01.003449 16194 consensus_queue.cc:984] T fdc69131212841d285ea88aadf8b6bd1 P 1b055b8db383436b94d48378a89755a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "41cd5177c3884ec29a257f84325c090b" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 33409 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 41, Last known committed idx: 40, Time since last communication: 0.000s I0507 06:34:01.007452 16194 raft_consensus.cc:2788] T fdc69131212841d285ea88aadf8b6bd1 P 1b055b8db383436b94d48378a89755a8 [term 2 LEADER]: Committing config change with OpId 2.41: config changed from index 40 to 41, 407e162251194eb69b048b59362a51bf (127.0.57.194) changed from NON_VOTER to VOTER. New config: { opid_index: 41 OBSOLETE_local: false peers { permanent_uuid: "1b055b8db383436b94d48378a89755a8" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 37465 } } peers { permanent_uuid: "41cd5177c3884ec29a257f84325c090b" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 33409 } } peers { permanent_uuid: "407e162251194eb69b048b59362a51bf" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 32837 } attrs { promote: false } } } I0507 06:34:01.008219 16135 raft_consensus.cc:2788] T fdc69131212841d285ea88aadf8b6bd1 P 407e162251194eb69b048b59362a51bf [term 2 FOLLOWER]: Committing config change with OpId 2.41: config changed from index 40 to 41, 407e162251194eb69b048b59362a51bf (127.0.57.194) changed from NON_VOTER to VOTER. New config: { opid_index: 41 OBSOLETE_local: false peers { permanent_uuid: "1b055b8db383436b94d48378a89755a8" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 37465 } } peers { permanent_uuid: "41cd5177c3884ec29a257f84325c090b" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 33409 } } peers { permanent_uuid: "407e162251194eb69b048b59362a51bf" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 32837 } attrs { promote: false } } } I0507 06:34:01.011334 15558 catalog_manager.cc:3988] T fdc69131212841d285ea88aadf8b6bd1 P 1b055b8db383436b94d48378a89755a8 reported cstate change: config changed from index 40 to 41, 407e162251194eb69b048b59362a51bf (127.0.57.194) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "1b055b8db383436b94d48378a89755a8" committed_config { opid_index: 41 OBSOLETE_local: false peers { permanent_uuid: "1b055b8db383436b94d48378a89755a8" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 37465 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "41cd5177c3884ec29a257f84325c090b" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 33409 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "407e162251194eb69b048b59362a51bf" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 32837 } attrs { promote: false } health_report { overall_health: HEALTHY } } } I0507 06:34:01.011688 15679 raft_consensus.cc:2788] T fdc69131212841d285ea88aadf8b6bd1 P 41cd5177c3884ec29a257f84325c090b [term 2 FOLLOWER]: Committing config change with OpId 2.41: config changed from index 40 to 41, 407e162251194eb69b048b59362a51bf (127.0.57.194) changed from NON_VOTER to VOTER. New config: { opid_index: 41 OBSOLETE_local: false peers { permanent_uuid: "1b055b8db383436b94d48378a89755a8" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 37465 } } peers { permanent_uuid: "41cd5177c3884ec29a257f84325c090b" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 33409 } } peers { permanent_uuid: "407e162251194eb69b048b59362a51bf" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 32837 } attrs { promote: false } } } Master Summary UUID | Address | Status ----------------------------------+--------------------+--------- 96510d60d2b748329ce1e8d69faaad41 | 127.0.57.254:34189 | HEALTHY Flag | Value | Tags | Master ------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+------------------------- ipki_ca_key_size | 1024 | experimental | all 1 server(s) checked ipki_server_key_size | 1024 | experimental | all 1 server(s) checked never_fsync | true | unsafe,advanced | all 1 server(s) checked raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked rpc_reuseport | true | experimental | all 1 server(s) checked rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked server_dump_info_format | pb | hidden | all 1 server(s) checked server_dump_info_path | /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_4.1557210500371863-231/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked Tablet Server Summary UUID | Address | Status | Location ----------------------------------+--------------------+---------+---------- 1b055b8db383436b94d48378a89755a8 | 127.0.57.195:37465 | HEALTHY | 407e162251194eb69b048b59362a51bf | 127.0.57.194:32837 | HEALTHY | 41cd5177c3884ec29a257f84325c090b | 127.0.57.193:33409 | HEALTHY | Tablet Server Location Summary Location | Count ----------+--------- | 3 Flag | Value | Tags | Tablet Server ------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------------------------------+------------------------- ipki_server_key_size | 1024 | experimental | all 3 server(s) checked local_ip_for_outbound_sockets | 127.0.57.193 | experimental | 127.0.57.193:33409 local_ip_for_outbound_sockets | 127.0.57.194 | experimental | 127.0.57.194:32837 local_ip_for_outbound_sockets | 127.0.57.195 | experimental | 127.0.57.195:37465 log_compression_codec | none | experimental | all 3 server(s) checked log_max_segments_to_retain | 3 | experimental,advanced,runtime | all 3 server(s) checked log_target_replay_size_mb | 1 | experimental | all 3 server(s) checked maintenance_manager_polling_interval_ms | 100 | hidden | all 3 server(s) checked max_cell_size_bytes | 1000000 | unsafe | all 3 server(s) checked never_fsync | true | unsafe,advanced | all 3 server(s) checked raft_prepare_replacement_before_eviction | true | experimental,advanced | all 3 server(s) checked rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked server_dump_info_format | pb | hidden | all 3 server(s) checked server_dump_info_path | /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_4.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.0.57.193:33409 server_dump_info_path | /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_4.1557210500371863-231/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.0.57.194:32837 server_dump_info_path | /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_4.1557210500371863-231/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.0.57.195:37465 Version Summary Version | Servers ------------------------+------------------------- 1.10.0-cdh6.x-SNAPSHOT | all 4 server(s) checked Tablet Summary Summary by table Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable -----------+----+---------+---------------+---------+------------+------------------+------------- TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0 Tablet Replica Count Summary Statistic | Replica Count ----------------+--------------- Minimum | 1 First Quartile | 1 Median | 1 Third Quartile | 1 Maximum | 1 Total Count Summary | Total Count ----------------+------------- Masters | 1 Tablet Servers | 3 Tables | 1 Tablets | 1 Replicas | 3 ================== Warnings: ================== Some masters have unsafe, experimental, or hidden flags set Some tablet servers have unsafe, experimental, or hidden flags set OK I0507 06:34:01.116300 231 log_verifier.cc:121] Checking tablet fdc69131212841d285ea88aadf8b6bd1 I0507 06:34:01.133991 231 log_verifier.cc:172] Verified matching terms for 18 ops in tablet fdc69131212841d285ea88aadf8b6bd1 I0507 06:34:01.135485 231 external_mini_cluster.cc:1172] Attempting to check leaks for /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver pid 15595 I0507 06:34:01.136773 15720 generic_service.cc:171] Checking for leaks (request via RPC) I0507 06:34:01.222951 231 external_mini_cluster.cc:1131] Killing /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver with pid 15595 I0507 06:34:01.243252 231 external_mini_cluster.cc:1172] Attempting to check leaks for /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver pid 16056 I0507 06:34:01.248831 16175 generic_service.cc:171] Checking for leaks (request via RPC) I0507 06:34:01.302211 231 external_mini_cluster.cc:1131] Killing /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver with pid 16056 I0507 06:34:01.313442 231 external_mini_cluster.cc:1172] Attempting to check leaks for /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver pid 15849 I0507 06:34:01.314476 15965 generic_service.cc:171] Checking for leaks (request via RPC) I0507 06:34:01.413136 231 external_mini_cluster.cc:1131] Killing /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver with pid 15849 I0507 06:34:01.431902 231 external_mini_cluster.cc:1172] Attempting to check leaks for /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-master pid 15534 I0507 06:34:01.433035 15590 generic_service.cc:171] Checking for leaks (request via RPC) I0507 06:34:01.521373 231 external_mini_cluster.cc:1131] Killing /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-master with pid 15534 [ OK ] ReplicaBehindWalGcThresholdITest.ReplicaReplacement/4 (8554 ms) [ RUN ] ReplicaBehindWalGcThresholdITest.ReplicaReplacement/5 I0507 06:34:01.549192 231 test_util.cc:212] Using random seed: 1728907644 I0507 06:34:01.550694 231 ts_itest-base.cc:112] Starting cluster with: I0507 06:34:01.550727 231 ts_itest-base.cc:113] -------------- I0507 06:34:01.550740 231 ts_itest-base.cc:114] 3 tablet servers I0507 06:34:01.550750 231 ts_itest-base.cc:115] 3 replicas per TS I0507 06:34:01.550761 231 ts_itest-base.cc:116] -------------- I0507 06:34:01.551831 231 external_mini_cluster.cc:933] Running /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-master /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-master --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_5.1557210500371863-231/raft_consensus-itest-cluster/master-0/wal --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_5.1557210500371863-231/raft_consensus-itest-cluster/master-0/data --block_manager=log --webserver_interface=localhost --ipki_ca_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.0.57.254:38831 --webserver_interface=127.0.57.254 --webserver_port=0 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_5.1557210500371863-231/raft_consensus-itest-cluster/master-0/logs --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_5.1557210500371863-231/raft_consensus-itest-cluster/master-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=true --rpc_reuseport=true with env {} WARNING: Logging before InitGoogleLogging() is written to STDERR W0507 06:34:01.753046 16239 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0507 06:34:01.753262 16239 flags.cc:404] Enabled unsafe flag: --never_fsync=true W0507 06:34:01.755681 16239 flags.cc:404] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true W0507 06:34:01.755743 16239 flags.cc:404] Enabled experimental flag: --ipki_ca_key_size=1024 W0507 06:34:01.755771 16239 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W0507 06:34:01.755810 16239 flags.cc:404] Enabled experimental flag: --tsk_num_rsa_bits=512 W0507 06:34:01.755851 16239 flags.cc:404] Enabled experimental flag: --rpc_reuseport=true I0507 06:34:01.760268 16239 master_main.cc:73] Master server non-default flags: --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_5.1557210500371863-231/raft_consensus-itest-cluster/master-0/data --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_5.1557210500371863-231/raft_consensus-itest-cluster/master-0/wal --ipki_ca_key_size=1024 --ipki_server_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.0.57.254:38831 --rpc_reuseport=true --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_5.1557210500371863-231/raft_consensus-itest-cluster/master-0/data/info.pb --webserver_interface=127.0.57.254 --webserver_port=0 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_5.1557210500371863-231/raft_consensus-itest-cluster/master-0/logs --logbuflevel=-1 --logtostderr=true Master server version: kudu 1.10.0-cdh6.x-SNAPSHOT revision 7d0ee26697926e87af9ef395008799577f6ab43f build type FASTDEBUG built by jenkins at 06 May 2019 22:44:21 PST on quasar-knmpfx-1.vpc.cloudera.com build id 322017 ASAN enabled I0507 06:34:01.766917 16239 system_ntp.cc:202] NTP initialized. Skew: 500ppm Current error: 195016us I0507 06:34:01.768126 16239 fs_manager.cc:263] Metadata directory not provided I0507 06:34:01.768194 16239 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0507 06:34:01.768491 16239 server_base.cc:433] Could not load existing FS layout: Not found: could not find a healthy instance file I0507 06:34:01.768586 16239 server_base.cc:434] Attempting to create new FS layout instead I0507 06:34:01.770974 16239 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_5.1557210500371863-231/raft_consensus-itest-cluster/master-0/data/instance: uuid: "3d5e6cea88e548a1b14d0e4a8455d204" format_stamp: "Formatted at 2019-05-07 06:34:01 on dist-test-slave-dist-test-slave-rqp6" I0507 06:34:01.771492 16239 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_5.1557210500371863-231/raft_consensus-itest-cluster/master-0/wal/instance: uuid: "3d5e6cea88e548a1b14d0e4a8455d204" format_stamp: "Formatted at 2019-05-07 06:34:01 on dist-test-slave-dist-test-slave-rqp6" I0507 06:34:01.778734 16239 fs_manager.cc:503] Time spent creating directory manager: real 0.007s user 0.002s sys 0.006s I0507 06:34:01.779808 16239 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0507 06:34:01.780354 16239 file_cache.cc:466] Constructed file cache lbm with capacity 419430 I0507 06:34:01.783934 16239 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s I0507 06:34:01.784029 16239 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_5.1557210500371863-231/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_5.1557210500371863-231/raft_consensus-itest-cluster/master-0/wal uuid: "3d5e6cea88e548a1b14d0e4a8455d204" format_stamp: "Formatted at 2019-05-07 06:34:01 on dist-test-slave-dist-test-slave-rqp6" I0507 06:34:01.784181 16239 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_5.1557210500371863-231/raft_consensus-itest-cluster/master-0/wal metadata directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_5.1557210500371863-231/raft_consensus-itest-cluster/master-0/wal 1 data directories: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_5.1557210500371863-231/raft_consensus-itest-cluster/master-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0507 06:34:01.812574 16239 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0507 06:34:01.828552 16239 rpc_server.cc:205] RPC server started. Bound to: 127.0.57.254:38831 I0507 06:34:01.829715 16239 webserver.cc:308] Webserver started at http://127.0.57.254:41563/ using document root and password file I0507 06:34:01.830418 16239 server_base.cc:608] Dumped server information to /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_5.1557210500371863-231/raft_consensus-itest-cluster/master-0/data/info.pb I0507 06:34:01.834365 16298 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0507 06:34:01.839918 231 external_mini_cluster.cc:995] Started /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-master as pid 16239 I0507 06:34:01.840603 231 external_mini_cluster.cc:933] Running /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_5.1557210500371863-231/raft_consensus-itest-cluster/ts-0/wal --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_5.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data --block_manager=log --rpc_bind_addresses=127.0.57.193:0 --local_ip_for_outbound_sockets=127.0.57.193 --webserver_interface=127.0.57.193 --webserver_port=0 --tserver_master_addrs=127.0.57.254:38831 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_5.1557210500371863-231/raft_consensus-itest-cluster/ts-0/logs --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_5.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=true --follower_unavailable_considered_failed_sec=5 --log_compression_codec=none --log_cache_size_limit_mb=1 --log_segment_size_mb=1 --log_async_preallocate_segments=false --log_min_segments_to_retain=1 --log_max_segments_to_retain=3 --maintenance_manager_polling_interval_ms=100 --log_target_replay_size_mb=1 --max_cell_size_bytes=1000000 with env {} I0507 06:34:01.846532 16298 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 3d5e6cea88e548a1b14d0e4a8455d204: Bootstrap starting. I0507 06:34:01.849632 16298 tablet_bootstrap.cc:590] T 00000000000000000000000000000000 P 3d5e6cea88e548a1b14d0e4a8455d204: No blocks or log segments found. Creating new log. I0507 06:34:01.850703 16298 log.cc:525] T 00000000000000000000000000000000 P 3d5e6cea88e548a1b14d0e4a8455d204: Log is configured to *not* fsync() on all Append() calls I0507 06:34:01.852608 16298 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 3d5e6cea88e548a1b14d0e4a8455d204: No bootstrap required, opened a new log I0507 06:34:01.859825 16298 raft_consensus.cc:340] T 00000000000000000000000000000000 P 3d5e6cea88e548a1b14d0e4a8455d204 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d5e6cea88e548a1b14d0e4a8455d204" member_type: VOTER } I0507 06:34:01.860126 16298 raft_consensus.cc:366] T 00000000000000000000000000000000 P 3d5e6cea88e548a1b14d0e4a8455d204 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0507 06:34:01.860208 16298 raft_consensus.cc:705] T 00000000000000000000000000000000 P 3d5e6cea88e548a1b14d0e4a8455d204 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d5e6cea88e548a1b14d0e4a8455d204, State: Initialized, Role: FOLLOWER I0507 06:34:01.860810 16298 consensus_queue.cc:229] T 00000000000000000000000000000000 P 3d5e6cea88e548a1b14d0e4a8455d204 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d5e6cea88e548a1b14d0e4a8455d204" member_type: VOTER } I0507 06:34:01.861002 16298 raft_consensus.cc:380] T 00000000000000000000000000000000 P 3d5e6cea88e548a1b14d0e4a8455d204 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0507 06:34:01.861080 16298 raft_consensus.cc:472] T 00000000000000000000000000000000 P 3d5e6cea88e548a1b14d0e4a8455d204 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0507 06:34:01.861207 16298 raft_consensus.cc:2886] T 00000000000000000000000000000000 P 3d5e6cea88e548a1b14d0e4a8455d204 [term 0 FOLLOWER]: Advancing to term 1 I0507 06:34:01.862970 16298 raft_consensus.cc:494] T 00000000000000000000000000000000 P 3d5e6cea88e548a1b14d0e4a8455d204 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d5e6cea88e548a1b14d0e4a8455d204" member_type: VOTER } I0507 06:34:01.863461 16298 leader_election.cc:310] T 00000000000000000000000000000000 P 3d5e6cea88e548a1b14d0e4a8455d204 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3d5e6cea88e548a1b14d0e4a8455d204; no voters: I0507 06:34:01.864725 16298 leader_election.cc:296] T 00000000000000000000000000000000 P 3d5e6cea88e548a1b14d0e4a8455d204 [CANDIDATE]: Term 1 election: Requested vote from peers I0507 06:34:01.865097 16301 raft_consensus.cc:2640] T 00000000000000000000000000000000 P 3d5e6cea88e548a1b14d0e4a8455d204 [term 1 FOLLOWER]: Leader election won for term 1 I0507 06:34:01.872648 16301 raft_consensus.cc:667] T 00000000000000000000000000000000 P 3d5e6cea88e548a1b14d0e4a8455d204 [term 1 LEADER]: Becoming Leader. State: Replica: 3d5e6cea88e548a1b14d0e4a8455d204, State: Running, Role: LEADER I0507 06:34:01.873159 16301 consensus_queue.cc:206] T 00000000000000000000000000000000 P 3d5e6cea88e548a1b14d0e4a8455d204 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d5e6cea88e548a1b14d0e4a8455d204" member_type: VOTER } I0507 06:34:01.877004 16303 sys_catalog.cc:337] T 00000000000000000000000000000000 P 3d5e6cea88e548a1b14d0e4a8455d204 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3d5e6cea88e548a1b14d0e4a8455d204. Latest consensus state: current_term: 1 leader_uuid: "3d5e6cea88e548a1b14d0e4a8455d204" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d5e6cea88e548a1b14d0e4a8455d204" member_type: VOTER } } I0507 06:34:01.877454 16303 sys_catalog.cc:340] T 00000000000000000000000000000000 P 3d5e6cea88e548a1b14d0e4a8455d204 [sys.catalog]: This master's current role is: LEADER I0507 06:34:01.877665 16298 sys_catalog.cc:337] T 00000000000000000000000000000000 P 3d5e6cea88e548a1b14d0e4a8455d204 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 1 leader_uuid: "3d5e6cea88e548a1b14d0e4a8455d204" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d5e6cea88e548a1b14d0e4a8455d204" member_type: VOTER } } I0507 06:34:01.877948 16298 sys_catalog.cc:340] T 00000000000000000000000000000000 P 3d5e6cea88e548a1b14d0e4a8455d204 [sys.catalog]: This master's current role is: LEADER I0507 06:34:01.878051 16302 sys_catalog.cc:337] T 00000000000000000000000000000000 P 3d5e6cea88e548a1b14d0e4a8455d204 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3d5e6cea88e548a1b14d0e4a8455d204" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d5e6cea88e548a1b14d0e4a8455d204" member_type: VOTER } } I0507 06:34:01.878302 16302 sys_catalog.cc:340] T 00000000000000000000000000000000 P 3d5e6cea88e548a1b14d0e4a8455d204 [sys.catalog]: This master's current role is: LEADER I0507 06:34:01.879390 16298 sys_catalog.cc:424] T 00000000000000000000000000000000 P 3d5e6cea88e548a1b14d0e4a8455d204 [sys.catalog]: configured and running, proceeding with master startup. I0507 06:34:01.879611 16307 catalog_manager.cc:1055] Loading table and tablet metadata into memory... I0507 06:34:01.887209 16307 catalog_manager.cc:1066] Initializing Kudu internal certificate authority... I0507 06:34:01.921608 16307 catalog_manager.cc:950] Generated new certificate authority record I0507 06:34:01.922917 16307 catalog_manager.cc:1075] Loading token signing keys... I0507 06:34:01.935195 16307 catalog_manager.cc:4292] T 00000000000000000000000000000000 P 3d5e6cea88e548a1b14d0e4a8455d204: Generated new TSK 0 WARNING: Logging before InitGoogleLogging() is written to STDERR W0507 06:34:02.017457 16300 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0507 06:34:02.017678 16300 flags.cc:404] Enabled unsafe flag: --max_cell_size_bytes=1000000 W0507 06:34:02.017720 16300 flags.cc:404] Enabled unsafe flag: --never_fsync=true W0507 06:34:02.019742 16300 flags.cc:404] Enabled experimental flag: --log_compression_codec=none W0507 06:34:02.019804 16300 flags.cc:404] Enabled experimental flag: --log_max_segments_to_retain=3 W0507 06:34:02.019843 16300 flags.cc:404] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true W0507 06:34:02.019909 16300 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W0507 06:34:02.019987 16300 flags.cc:404] Enabled experimental flag: --log_target_replay_size_mb=1 W0507 06:34:02.020051 16300 flags.cc:404] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.57.193 I0507 06:34:02.037183 16300 tablet_server_main.cc:78] Tablet server non-default flags: --follower_unavailable_considered_failed_sec=5 --log_compression_codec=none --log_max_segments_to_retain=3 --log_cache_size_limit_mb=1 --log_async_preallocate_segments=false --log_segment_size_mb=1 --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_5.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_5.1557210500371863-231/raft_consensus-itest-cluster/ts-0/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.0.57.193:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_5.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data/info.pb --webserver_interface=127.0.57.193 --webserver_port=0 --max_cell_size_bytes=1000000 --tserver_master_addrs=127.0.57.254:38831 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --log_target_replay_size_mb=1 --maintenance_manager_polling_interval_ms=100 --enable_minidumps=false --local_ip_for_outbound_sockets=127.0.57.193 --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_5.1557210500371863-231/raft_consensus-itest-cluster/ts-0/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.10.0-cdh6.x-SNAPSHOT revision 7d0ee26697926e87af9ef395008799577f6ab43f build type FASTDEBUG built by jenkins at 06 May 2019 22:44:21 PST on quasar-knmpfx-1.vpc.cloudera.com build id 322017 ASAN enabled I0507 06:34:02.045230 16300 system_ntp.cc:202] NTP initialized. Skew: 500ppm Current error: 195516us I0507 06:34:02.045904 16300 fs_manager.cc:263] Metadata directory not provided I0507 06:34:02.045969 16300 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0507 06:34:02.046233 16300 server_base.cc:433] Could not load existing FS layout: Not found: could not find a healthy instance file I0507 06:34:02.046303 16300 server_base.cc:434] Attempting to create new FS layout instead I0507 06:34:02.048609 16300 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_5.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data/instance: uuid: "40616cbd3b6a40bd860286f0be106119" format_stamp: "Formatted at 2019-05-07 06:34:02 on dist-test-slave-dist-test-slave-rqp6" I0507 06:34:02.049221 16300 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_5.1557210500371863-231/raft_consensus-itest-cluster/ts-0/wal/instance: uuid: "40616cbd3b6a40bd860286f0be106119" format_stamp: "Formatted at 2019-05-07 06:34:02 on dist-test-slave-dist-test-slave-rqp6" I0507 06:34:02.058578 16300 fs_manager.cc:503] Time spent creating directory manager: real 0.009s user 0.006s sys 0.002s I0507 06:34:02.059664 16300 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0507 06:34:02.060276 16300 file_cache.cc:466] Constructed file cache lbm with capacity 419430 I0507 06:34:02.063730 16300 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.000s sys 0.000s I0507 06:34:02.063828 16300 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_5.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_5.1557210500371863-231/raft_consensus-itest-cluster/ts-0/wal uuid: "40616cbd3b6a40bd860286f0be106119" format_stamp: "Formatted at 2019-05-07 06:34:02 on dist-test-slave-dist-test-slave-rqp6" I0507 06:34:02.064004 16300 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_5.1557210500371863-231/raft_consensus-itest-cluster/ts-0/wal metadata directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_5.1557210500371863-231/raft_consensus-itest-cluster/ts-0/wal 1 data directories: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_5.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0507 06:34:02.109773 16300 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0507 06:34:02.111483 16300 ts_tablet_manager.cc:356] Loaded tablet metadata (0 total tablets, 0 live tablets) I0507 06:34:02.111564 16300 ts_tablet_manager.cc:376] Registered 0 tablets I0507 06:34:02.200312 16300 rpc_server.cc:205] RPC server started. Bound to: 127.0.57.193:43553 I0507 06:34:02.201509 16300 webserver.cc:308] Webserver started at http://127.0.57.193:40827/ using document root and password file I0507 06:34:02.202174 16300 server_base.cc:608] Dumped server information to /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_5.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data/info.pb I0507 06:34:02.209782 231 external_mini_cluster.cc:995] Started /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver as pid 16300 I0507 06:34:02.210626 231 external_mini_cluster.cc:933] Running /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_5.1557210500371863-231/raft_consensus-itest-cluster/ts-1/wal --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_5.1557210500371863-231/raft_consensus-itest-cluster/ts-1/data --block_manager=log --rpc_bind_addresses=127.0.57.194:0 --local_ip_for_outbound_sockets=127.0.57.194 --webserver_interface=127.0.57.194 --webserver_port=0 --tserver_master_addrs=127.0.57.254:38831 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_5.1557210500371863-231/raft_consensus-itest-cluster/ts-1/logs --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_5.1557210500371863-231/raft_consensus-itest-cluster/ts-1/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=true --follower_unavailable_considered_failed_sec=5 --log_compression_codec=none --log_cache_size_limit_mb=1 --log_segment_size_mb=1 --log_async_preallocate_segments=false --log_min_segments_to_retain=1 --log_max_segments_to_retain=3 --maintenance_manager_polling_interval_ms=100 --log_target_replay_size_mb=1 --max_cell_size_bytes=1000000 with env {} I0507 06:34:02.223757 16429 heartbeater.cc:345] Connected to a master server at 127.0.57.254:38831 I0507 06:34:02.224032 16429 heartbeater.cc:426] Registering TS with master... I0507 06:34:02.224769 16429 heartbeater.cc:475] Master 127.0.57.254:38831 requested a full tablet report, sending... I0507 06:34:02.226611 16265 ts_manager.cc:159] Registered new tserver with Master: 40616cbd3b6a40bd860286f0be106119 (127.0.57.193:43553) I0507 06:34:02.228193 16265 master_service.cc:273] Signed X509 certificate for tserver {username='slave'} at 127.0.57.193:41245 WARNING: Logging before InitGoogleLogging() is written to STDERR W0507 06:34:02.390378 16433 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0507 06:34:02.390888 16433 flags.cc:404] Enabled unsafe flag: --max_cell_size_bytes=1000000 W0507 06:34:02.390980 16433 flags.cc:404] Enabled unsafe flag: --never_fsync=true W0507 06:34:02.393952 16433 flags.cc:404] Enabled experimental flag: --log_compression_codec=none W0507 06:34:02.394035 16433 flags.cc:404] Enabled experimental flag: --log_max_segments_to_retain=3 W0507 06:34:02.394102 16433 flags.cc:404] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true W0507 06:34:02.394167 16433 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W0507 06:34:02.394289 16433 flags.cc:404] Enabled experimental flag: --log_target_replay_size_mb=1 W0507 06:34:02.394351 16433 flags.cc:404] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.57.194 I0507 06:34:02.398754 16433 tablet_server_main.cc:78] Tablet server non-default flags: --follower_unavailable_considered_failed_sec=5 --log_compression_codec=none --log_max_segments_to_retain=3 --log_cache_size_limit_mb=1 --log_async_preallocate_segments=false --log_segment_size_mb=1 --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_5.1557210500371863-231/raft_consensus-itest-cluster/ts-1/data --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_5.1557210500371863-231/raft_consensus-itest-cluster/ts-1/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.0.57.194:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_5.1557210500371863-231/raft_consensus-itest-cluster/ts-1/data/info.pb --webserver_interface=127.0.57.194 --webserver_port=0 --max_cell_size_bytes=1000000 --tserver_master_addrs=127.0.57.254:38831 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --log_target_replay_size_mb=1 --maintenance_manager_polling_interval_ms=100 --enable_minidumps=false --local_ip_for_outbound_sockets=127.0.57.194 --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_5.1557210500371863-231/raft_consensus-itest-cluster/ts-1/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.10.0-cdh6.x-SNAPSHOT revision 7d0ee26697926e87af9ef395008799577f6ab43f build type FASTDEBUG built by jenkins at 06 May 2019 22:44:21 PST on quasar-knmpfx-1.vpc.cloudera.com build id 322017 ASAN enabled I0507 06:34:02.406388 16433 system_ntp.cc:202] NTP initialized. Skew: 500ppm Current error: 195516us I0507 06:34:02.407073 16433 fs_manager.cc:263] Metadata directory not provided I0507 06:34:02.407135 16433 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0507 06:34:02.407418 16433 server_base.cc:433] Could not load existing FS layout: Not found: could not find a healthy instance file I0507 06:34:02.407482 16433 server_base.cc:434] Attempting to create new FS layout instead I0507 06:34:02.409957 16433 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_5.1557210500371863-231/raft_consensus-itest-cluster/ts-1/data/instance: uuid: "58f3e2c789b64bb0a81cc10c779b95e3" format_stamp: "Formatted at 2019-05-07 06:34:02 on dist-test-slave-dist-test-slave-rqp6" I0507 06:34:02.410506 16433 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_5.1557210500371863-231/raft_consensus-itest-cluster/ts-1/wal/instance: uuid: "58f3e2c789b64bb0a81cc10c779b95e3" format_stamp: "Formatted at 2019-05-07 06:34:02 on dist-test-slave-dist-test-slave-rqp6" I0507 06:34:02.416431 16433 fs_manager.cc:503] Time spent creating directory manager: real 0.006s user 0.002s sys 0.004s I0507 06:34:02.417461 16433 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0507 06:34:02.418035 16433 file_cache.cc:466] Constructed file cache lbm with capacity 419430 I0507 06:34:02.420881 16433 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.000s sys 0.000s I0507 06:34:02.420964 16433 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_5.1557210500371863-231/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_5.1557210500371863-231/raft_consensus-itest-cluster/ts-1/wal uuid: "58f3e2c789b64bb0a81cc10c779b95e3" format_stamp: "Formatted at 2019-05-07 06:34:02 on dist-test-slave-dist-test-slave-rqp6" I0507 06:34:02.421134 16433 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_5.1557210500371863-231/raft_consensus-itest-cluster/ts-1/wal metadata directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_5.1557210500371863-231/raft_consensus-itest-cluster/ts-1/wal 1 data directories: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_5.1557210500371863-231/raft_consensus-itest-cluster/ts-1/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0507 06:34:02.477283 16433 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0507 06:34:02.479128 16433 ts_tablet_manager.cc:356] Loaded tablet metadata (0 total tablets, 0 live tablets) I0507 06:34:02.479243 16433 ts_tablet_manager.cc:376] Registered 0 tablets I0507 06:34:02.501843 16433 rpc_server.cc:205] RPC server started. Bound to: 127.0.57.194:38173 I0507 06:34:02.502982 16433 webserver.cc:308] Webserver started at http://127.0.57.194:39053/ using document root and password file I0507 06:34:02.503667 16433 server_base.cc:608] Dumped server information to /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_5.1557210500371863-231/raft_consensus-itest-cluster/ts-1/data/info.pb I0507 06:34:02.507755 231 external_mini_cluster.cc:995] Started /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver as pid 16433 I0507 06:34:02.508568 231 external_mini_cluster.cc:933] Running /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_5.1557210500371863-231/raft_consensus-itest-cluster/ts-2/wal --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_5.1557210500371863-231/raft_consensus-itest-cluster/ts-2/data --block_manager=log --rpc_bind_addresses=127.0.57.195:0 --local_ip_for_outbound_sockets=127.0.57.195 --webserver_interface=127.0.57.195 --webserver_port=0 --tserver_master_addrs=127.0.57.254:38831 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_5.1557210500371863-231/raft_consensus-itest-cluster/ts-2/logs --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_5.1557210500371863-231/raft_consensus-itest-cluster/ts-2/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=true --follower_unavailable_considered_failed_sec=5 --log_compression_codec=none --log_cache_size_limit_mb=1 --log_segment_size_mb=1 --log_async_preallocate_segments=false --log_min_segments_to_retain=1 --log_max_segments_to_retain=3 --maintenance_manager_polling_interval_ms=100 --log_target_replay_size_mb=1 --max_cell_size_bytes=1000000 with env {} I0507 06:34:02.517748 16552 heartbeater.cc:345] Connected to a master server at 127.0.57.254:38831 I0507 06:34:02.517958 16552 heartbeater.cc:426] Registering TS with master... I0507 06:34:02.518497 16552 heartbeater.cc:475] Master 127.0.57.254:38831 requested a full tablet report, sending... I0507 06:34:02.519635 16265 ts_manager.cc:159] Registered new tserver with Master: 58f3e2c789b64bb0a81cc10c779b95e3 (127.0.57.194:38173) I0507 06:34:02.520514 16265 master_service.cc:273] Signed X509 certificate for tserver {username='slave'} at 127.0.57.194:39933 WARNING: Logging before InitGoogleLogging() is written to STDERR W0507 06:34:02.685261 16555 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0507 06:34:02.685472 16555 flags.cc:404] Enabled unsafe flag: --max_cell_size_bytes=1000000 W0507 06:34:02.685515 16555 flags.cc:404] Enabled unsafe flag: --never_fsync=true W0507 06:34:02.687531 16555 flags.cc:404] Enabled experimental flag: --log_compression_codec=none W0507 06:34:02.687590 16555 flags.cc:404] Enabled experimental flag: --log_max_segments_to_retain=3 W0507 06:34:02.687636 16555 flags.cc:404] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true W0507 06:34:02.687681 16555 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W0507 06:34:02.687759 16555 flags.cc:404] Enabled experimental flag: --log_target_replay_size_mb=1 W0507 06:34:02.687808 16555 flags.cc:404] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.57.195 I0507 06:34:02.691869 16555 tablet_server_main.cc:78] Tablet server non-default flags: --follower_unavailable_considered_failed_sec=5 --log_compression_codec=none --log_max_segments_to_retain=3 --log_cache_size_limit_mb=1 --log_async_preallocate_segments=false --log_segment_size_mb=1 --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_5.1557210500371863-231/raft_consensus-itest-cluster/ts-2/data --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_5.1557210500371863-231/raft_consensus-itest-cluster/ts-2/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.0.57.195:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_5.1557210500371863-231/raft_consensus-itest-cluster/ts-2/data/info.pb --webserver_interface=127.0.57.195 --webserver_port=0 --max_cell_size_bytes=1000000 --tserver_master_addrs=127.0.57.254:38831 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --log_target_replay_size_mb=1 --maintenance_manager_polling_interval_ms=100 --enable_minidumps=false --local_ip_for_outbound_sockets=127.0.57.195 --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_5.1557210500371863-231/raft_consensus-itest-cluster/ts-2/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.10.0-cdh6.x-SNAPSHOT revision 7d0ee26697926e87af9ef395008799577f6ab43f build type FASTDEBUG built by jenkins at 06 May 2019 22:44:21 PST on quasar-knmpfx-1.vpc.cloudera.com build id 322017 ASAN enabled I0507 06:34:02.699465 16555 system_ntp.cc:202] NTP initialized. Skew: 500ppm Current error: 195516us I0507 06:34:02.700120 16555 fs_manager.cc:263] Metadata directory not provided I0507 06:34:02.700178 16555 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0507 06:34:02.700454 16555 server_base.cc:433] Could not load existing FS layout: Not found: could not find a healthy instance file I0507 06:34:02.700521 16555 server_base.cc:434] Attempting to create new FS layout instead I0507 06:34:02.702939 16555 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_5.1557210500371863-231/raft_consensus-itest-cluster/ts-2/data/instance: uuid: "48ea1474af5442249334996e0babce18" format_stamp: "Formatted at 2019-05-07 06:34:02 on dist-test-slave-dist-test-slave-rqp6" I0507 06:34:02.703516 16555 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_5.1557210500371863-231/raft_consensus-itest-cluster/ts-2/wal/instance: uuid: "48ea1474af5442249334996e0babce18" format_stamp: "Formatted at 2019-05-07 06:34:02 on dist-test-slave-dist-test-slave-rqp6" I0507 06:34:02.709894 16555 fs_manager.cc:503] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s I0507 06:34:02.710964 16555 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0507 06:34:02.711537 16555 file_cache.cc:466] Constructed file cache lbm with capacity 419430 I0507 06:34:02.714424 16555 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.000s sys 0.000s I0507 06:34:02.714511 16555 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_5.1557210500371863-231/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_5.1557210500371863-231/raft_consensus-itest-cluster/ts-2/wal uuid: "48ea1474af5442249334996e0babce18" format_stamp: "Formatted at 2019-05-07 06:34:02 on dist-test-slave-dist-test-slave-rqp6" I0507 06:34:02.714674 16555 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_5.1557210500371863-231/raft_consensus-itest-cluster/ts-2/wal metadata directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_5.1557210500371863-231/raft_consensus-itest-cluster/ts-2/wal 1 data directories: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_5.1557210500371863-231/raft_consensus-itest-cluster/ts-2/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0507 06:34:02.741834 16555 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0507 06:34:02.743455 16555 ts_tablet_manager.cc:356] Loaded tablet metadata (0 total tablets, 0 live tablets) I0507 06:34:02.743526 16555 ts_tablet_manager.cc:376] Registered 0 tablets I0507 06:34:02.767135 16555 rpc_server.cc:205] RPC server started. Bound to: 127.0.57.195:37195 I0507 06:34:02.768256 16555 webserver.cc:308] Webserver started at http://127.0.57.195:36047/ using document root and password file I0507 06:34:02.768939 16555 server_base.cc:608] Dumped server information to /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_5.1557210500371863-231/raft_consensus-itest-cluster/ts-2/data/info.pb I0507 06:34:02.770153 231 external_mini_cluster.cc:995] Started /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver as pid 16555 I0507 06:34:02.782763 16674 heartbeater.cc:345] Connected to a master server at 127.0.57.254:38831 I0507 06:34:02.782965 16674 heartbeater.cc:426] Registering TS with master... I0507 06:34:02.783483 16674 heartbeater.cc:475] Master 127.0.57.254:38831 requested a full tablet report, sending... I0507 06:34:02.784489 16265 ts_manager.cc:159] Registered new tserver with Master: 48ea1474af5442249334996e0babce18 (127.0.57.195:37195) I0507 06:34:02.785432 231 external_mini_cluster.cc:623] 3 TS(s) registered with all masters I0507 06:34:02.785651 16265 master_service.cc:273] Signed X509 certificate for tserver {username='slave'} at 127.0.57.195:48093 I0507 06:34:02.796149 16265 catalog_manager.cc:1415] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53874: name: "TestTable" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 3 split_rows_range_bounds { } partition_schema { range_schema { columns { name: "key" } } } W0507 06:34:02.800866 16265 catalog_manager.cc:1574] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestTable in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers. I0507 06:34:02.834892 16366 tablet_service.cc:1016] Processing CreateTablet for tablet 1119750e874a4ae1b742612a6803ea81 (table=TestTable [id=4efac679a2544c17946c9f4b94d0b144]), partition=RANGE (key) PARTITION UNBOUNDED I0507 06:34:02.834892 16489 tablet_service.cc:1016] Processing CreateTablet for tablet 1119750e874a4ae1b742612a6803ea81 (table=TestTable [id=4efac679a2544c17946c9f4b94d0b144]), partition=RANGE (key) PARTITION UNBOUNDED I0507 06:34:02.835880 16366 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet 1119750e874a4ae1b742612a6803ea81. 1 dirs total, 0 dirs full, 0 dirs failed I0507 06:34:02.835880 16489 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet 1119750e874a4ae1b742612a6803ea81. 1 dirs total, 0 dirs full, 0 dirs failed I0507 06:34:02.835732 16611 tablet_service.cc:1016] Processing CreateTablet for tablet 1119750e874a4ae1b742612a6803ea81 (table=TestTable [id=4efac679a2544c17946c9f4b94d0b144]), partition=RANGE (key) PARTITION UNBOUNDED I0507 06:34:02.836416 16611 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet 1119750e874a4ae1b742612a6803ea81. 1 dirs total, 0 dirs full, 0 dirs failed I0507 06:34:02.846011 16693 tablet_bootstrap.cc:439] T 1119750e874a4ae1b742612a6803ea81 P 48ea1474af5442249334996e0babce18: Bootstrap starting. I0507 06:34:02.848249 16693 tablet_bootstrap.cc:590] T 1119750e874a4ae1b742612a6803ea81 P 48ea1474af5442249334996e0babce18: No blocks or log segments found. Creating new log. I0507 06:34:02.849059 16695 tablet_bootstrap.cc:439] T 1119750e874a4ae1b742612a6803ea81 P 40616cbd3b6a40bd860286f0be106119: Bootstrap starting. I0507 06:34:02.851598 16695 tablet_bootstrap.cc:590] T 1119750e874a4ae1b742612a6803ea81 P 40616cbd3b6a40bd860286f0be106119: No blocks or log segments found. Creating new log. I0507 06:34:02.853138 16694 tablet_bootstrap.cc:439] T 1119750e874a4ae1b742612a6803ea81 P 58f3e2c789b64bb0a81cc10c779b95e3: Bootstrap starting. I0507 06:34:02.855798 16694 tablet_bootstrap.cc:590] T 1119750e874a4ae1b742612a6803ea81 P 58f3e2c789b64bb0a81cc10c779b95e3: No blocks or log segments found. Creating new log. I0507 06:34:02.857333 16693 log.cc:525] T 1119750e874a4ae1b742612a6803ea81 P 48ea1474af5442249334996e0babce18: Log is configured to *not* fsync() on all Append() calls I0507 06:34:02.858475 16694 log.cc:525] T 1119750e874a4ae1b742612a6803ea81 P 58f3e2c789b64bb0a81cc10c779b95e3: Log is configured to *not* fsync() on all Append() calls I0507 06:34:02.858794 16695 log.cc:525] T 1119750e874a4ae1b742612a6803ea81 P 40616cbd3b6a40bd860286f0be106119: Log is configured to *not* fsync() on all Append() calls I0507 06:34:02.860448 16693 tablet_bootstrap.cc:439] T 1119750e874a4ae1b742612a6803ea81 P 48ea1474af5442249334996e0babce18: No bootstrap required, opened a new log I0507 06:34:02.860733 16693 ts_tablet_manager.cc:1053] T 1119750e874a4ae1b742612a6803ea81 P 48ea1474af5442249334996e0babce18: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.000s I0507 06:34:02.860767 16694 tablet_bootstrap.cc:439] T 1119750e874a4ae1b742612a6803ea81 P 58f3e2c789b64bb0a81cc10c779b95e3: No bootstrap required, opened a new log I0507 06:34:02.860967 16694 ts_tablet_manager.cc:1053] T 1119750e874a4ae1b742612a6803ea81 P 58f3e2c789b64bb0a81cc10c779b95e3: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s I0507 06:34:02.865694 16695 tablet_bootstrap.cc:439] T 1119750e874a4ae1b742612a6803ea81 P 40616cbd3b6a40bd860286f0be106119: No bootstrap required, opened a new log I0507 06:34:02.865985 16695 ts_tablet_manager.cc:1053] T 1119750e874a4ae1b742612a6803ea81 P 40616cbd3b6a40bd860286f0be106119: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.000s I0507 06:34:02.866562 16694 raft_consensus.cc:340] T 1119750e874a4ae1b742612a6803ea81 P 58f3e2c789b64bb0a81cc10c779b95e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58f3e2c789b64bb0a81cc10c779b95e3" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 38173 } } peers { permanent_uuid: "40616cbd3b6a40bd860286f0be106119" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 43553 } } peers { permanent_uuid: "48ea1474af5442249334996e0babce18" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 37195 } } I0507 06:34:02.866565 16693 raft_consensus.cc:340] T 1119750e874a4ae1b742612a6803ea81 P 48ea1474af5442249334996e0babce18 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58f3e2c789b64bb0a81cc10c779b95e3" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 38173 } } peers { permanent_uuid: "40616cbd3b6a40bd860286f0be106119" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 43553 } } peers { permanent_uuid: "48ea1474af5442249334996e0babce18" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 37195 } } I0507 06:34:02.866905 16694 raft_consensus.cc:366] T 1119750e874a4ae1b742612a6803ea81 P 58f3e2c789b64bb0a81cc10c779b95e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0507 06:34:02.866981 16693 raft_consensus.cc:366] T 1119750e874a4ae1b742612a6803ea81 P 48ea1474af5442249334996e0babce18 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0507 06:34:02.866981 16694 raft_consensus.cc:705] T 1119750e874a4ae1b742612a6803ea81 P 58f3e2c789b64bb0a81cc10c779b95e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58f3e2c789b64bb0a81cc10c779b95e3, State: Initialized, Role: FOLLOWER I0507 06:34:02.867051 16693 raft_consensus.cc:705] T 1119750e874a4ae1b742612a6803ea81 P 48ea1474af5442249334996e0babce18 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48ea1474af5442249334996e0babce18, State: Initialized, Role: FOLLOWER I0507 06:34:02.867475 16694 consensus_queue.cc:229] T 1119750e874a4ae1b742612a6803ea81 P 58f3e2c789b64bb0a81cc10c779b95e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58f3e2c789b64bb0a81cc10c779b95e3" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 38173 } } peers { permanent_uuid: "40616cbd3b6a40bd860286f0be106119" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 43553 } } peers { permanent_uuid: "48ea1474af5442249334996e0babce18" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 37195 } } I0507 06:34:02.867583 16693 consensus_queue.cc:229] T 1119750e874a4ae1b742612a6803ea81 P 48ea1474af5442249334996e0babce18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58f3e2c789b64bb0a81cc10c779b95e3" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 38173 } } peers { permanent_uuid: "40616cbd3b6a40bd860286f0be106119" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 43553 } } peers { permanent_uuid: "48ea1474af5442249334996e0babce18" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 37195 } } I0507 06:34:02.868325 16552 heartbeater.cc:467] Master 127.0.57.254:38831 was elected leader, sending a full tablet report... I0507 06:34:02.868598 16674 heartbeater.cc:467] Master 127.0.57.254:38831 was elected leader, sending a full tablet report... I0507 06:34:02.869295 16694 ts_tablet_manager.cc:1081] T 1119750e874a4ae1b742612a6803ea81 P 58f3e2c789b64bb0a81cc10c779b95e3: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s I0507 06:34:02.870694 16693 ts_tablet_manager.cc:1081] T 1119750e874a4ae1b742612a6803ea81 P 48ea1474af5442249334996e0babce18: Time spent starting tablet: real 0.010s user 0.008s sys 0.000s W0507 06:34:02.873704 16675 tablet.cc:1786] T 1119750e874a4ae1b742612a6803ea81 P 48ea1474af5442249334996e0babce18: Can't schedule compaction. Clean time has not been advanced past its initial value. I0507 06:34:02.873543 16695 raft_consensus.cc:340] T 1119750e874a4ae1b742612a6803ea81 P 40616cbd3b6a40bd860286f0be106119 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58f3e2c789b64bb0a81cc10c779b95e3" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 38173 } } peers { permanent_uuid: "40616cbd3b6a40bd860286f0be106119" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 43553 } } peers { permanent_uuid: "48ea1474af5442249334996e0babce18" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 37195 } } I0507 06:34:02.873924 16695 raft_consensus.cc:366] T 1119750e874a4ae1b742612a6803ea81 P 40616cbd3b6a40bd860286f0be106119 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0507 06:34:02.873980 16695 raft_consensus.cc:705] T 1119750e874a4ae1b742612a6803ea81 P 40616cbd3b6a40bd860286f0be106119 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40616cbd3b6a40bd860286f0be106119, State: Initialized, Role: FOLLOWER I0507 06:34:02.874404 16695 consensus_queue.cc:229] T 1119750e874a4ae1b742612a6803ea81 P 40616cbd3b6a40bd860286f0be106119 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58f3e2c789b64bb0a81cc10c779b95e3" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 38173 } } peers { permanent_uuid: "40616cbd3b6a40bd860286f0be106119" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 43553 } } peers { permanent_uuid: "48ea1474af5442249334996e0babce18" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 37195 } } I0507 06:34:02.875169 16429 heartbeater.cc:467] Master 127.0.57.254:38831 was elected leader, sending a full tablet report... I0507 06:34:02.875488 16695 ts_tablet_manager.cc:1081] T 1119750e874a4ae1b742612a6803ea81 P 40616cbd3b6a40bd860286f0be106119: Time spent starting tablet: real 0.009s user 0.005s sys 0.005s W0507 06:34:02.905411 16430 tablet.cc:1786] T 1119750e874a4ae1b742612a6803ea81 P 40616cbd3b6a40bd860286f0be106119: Can't schedule compaction. Clean time has not been advanced past its initial value. W0507 06:34:02.905875 16553 tablet.cc:1786] T 1119750e874a4ae1b742612a6803ea81 P 58f3e2c789b64bb0a81cc10c779b95e3: Can't schedule compaction. Clean time has not been advanced past its initial value. I0507 06:34:03.124487 16701 raft_consensus.cc:472] T 1119750e874a4ae1b742612a6803ea81 P 40616cbd3b6a40bd860286f0be106119 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0507 06:34:03.124737 16701 raft_consensus.cc:494] T 1119750e874a4ae1b742612a6803ea81 P 40616cbd3b6a40bd860286f0be106119 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58f3e2c789b64bb0a81cc10c779b95e3" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 38173 } } peers { permanent_uuid: "40616cbd3b6a40bd860286f0be106119" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 43553 } } peers { permanent_uuid: "48ea1474af5442249334996e0babce18" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 37195 } } I0507 06:34:03.125977 16701 leader_election.cc:296] T 1119750e874a4ae1b742612a6803ea81 P 40616cbd3b6a40bd860286f0be106119 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 58f3e2c789b64bb0a81cc10c779b95e3 (127.0.57.194:38173), 48ea1474af5442249334996e0babce18 (127.0.57.195:37195) I0507 06:34:03.132829 16509 tablet_service.cc:1286] Received RequestConsensusVote() RPC: tablet_id: "1119750e874a4ae1b742612a6803ea81" candidate_uuid: "40616cbd3b6a40bd860286f0be106119" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58f3e2c789b64bb0a81cc10c779b95e3" is_pre_election: true I0507 06:34:03.132834 16631 tablet_service.cc:1286] Received RequestConsensusVote() RPC: tablet_id: "1119750e874a4ae1b742612a6803ea81" candidate_uuid: "40616cbd3b6a40bd860286f0be106119" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48ea1474af5442249334996e0babce18" is_pre_election: true I0507 06:34:03.133167 16509 raft_consensus.cc:2346] T 1119750e874a4ae1b742612a6803ea81 P 58f3e2c789b64bb0a81cc10c779b95e3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 40616cbd3b6a40bd860286f0be106119 in term 0. I0507 06:34:03.133249 16631 raft_consensus.cc:2346] T 1119750e874a4ae1b742612a6803ea81 P 48ea1474af5442249334996e0babce18 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 40616cbd3b6a40bd860286f0be106119 in term 0. I0507 06:34:03.133785 16320 leader_election.cc:310] T 1119750e874a4ae1b742612a6803ea81 P 40616cbd3b6a40bd860286f0be106119 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 40616cbd3b6a40bd860286f0be106119, 48ea1474af5442249334996e0babce18; no voters: I0507 06:34:03.134274 16701 raft_consensus.cc:2640] T 1119750e874a4ae1b742612a6803ea81 P 40616cbd3b6a40bd860286f0be106119 [term 0 FOLLOWER]: Leader pre-election won for term 1 I0507 06:34:03.134356 16701 raft_consensus.cc:472] T 1119750e874a4ae1b742612a6803ea81 P 40616cbd3b6a40bd860286f0be106119 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0507 06:34:03.134407 16701 raft_consensus.cc:2886] T 1119750e874a4ae1b742612a6803ea81 P 40616cbd3b6a40bd860286f0be106119 [term 0 FOLLOWER]: Advancing to term 1 I0507 06:34:03.136116 16701 raft_consensus.cc:494] T 1119750e874a4ae1b742612a6803ea81 P 40616cbd3b6a40bd860286f0be106119 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58f3e2c789b64bb0a81cc10c779b95e3" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 38173 } } peers { permanent_uuid: "40616cbd3b6a40bd860286f0be106119" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 43553 } } peers { permanent_uuid: "48ea1474af5442249334996e0babce18" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 37195 } } I0507 06:34:03.136662 16701 leader_election.cc:296] T 1119750e874a4ae1b742612a6803ea81 P 40616cbd3b6a40bd860286f0be106119 [CANDIDATE]: Term 1 election: Requested vote from peers 58f3e2c789b64bb0a81cc10c779b95e3 (127.0.57.194:38173), 48ea1474af5442249334996e0babce18 (127.0.57.195:37195) I0507 06:34:03.137197 16509 tablet_service.cc:1286] Received RequestConsensusVote() RPC: tablet_id: "1119750e874a4ae1b742612a6803ea81" candidate_uuid: "40616cbd3b6a40bd860286f0be106119" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58f3e2c789b64bb0a81cc10c779b95e3" I0507 06:34:03.137197 16631 tablet_service.cc:1286] Received RequestConsensusVote() RPC: tablet_id: "1119750e874a4ae1b742612a6803ea81" candidate_uuid: "40616cbd3b6a40bd860286f0be106119" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48ea1474af5442249334996e0babce18" I0507 06:34:03.137390 16509 raft_consensus.cc:2886] T 1119750e874a4ae1b742612a6803ea81 P 58f3e2c789b64bb0a81cc10c779b95e3 [term 0 FOLLOWER]: Advancing to term 1 I0507 06:34:03.137495 16631 raft_consensus.cc:2886] T 1119750e874a4ae1b742612a6803ea81 P 48ea1474af5442249334996e0babce18 [term 0 FOLLOWER]: Advancing to term 1 I0507 06:34:03.139859 16509 raft_consensus.cc:2346] T 1119750e874a4ae1b742612a6803ea81 P 58f3e2c789b64bb0a81cc10c779b95e3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 40616cbd3b6a40bd860286f0be106119 in term 1. I0507 06:34:03.139858 16631 raft_consensus.cc:2346] T 1119750e874a4ae1b742612a6803ea81 P 48ea1474af5442249334996e0babce18 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 40616cbd3b6a40bd860286f0be106119 in term 1. I0507 06:34:03.140296 16317 leader_election.cc:310] T 1119750e874a4ae1b742612a6803ea81 P 40616cbd3b6a40bd860286f0be106119 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 40616cbd3b6a40bd860286f0be106119, 58f3e2c789b64bb0a81cc10c779b95e3; no voters: I0507 06:34:03.140619 16701 raft_consensus.cc:2640] T 1119750e874a4ae1b742612a6803ea81 P 40616cbd3b6a40bd860286f0be106119 [term 1 FOLLOWER]: Leader election won for term 1 I0507 06:34:03.140995 16701 raft_consensus.cc:667] T 1119750e874a4ae1b742612a6803ea81 P 40616cbd3b6a40bd860286f0be106119 [term 1 LEADER]: Becoming Leader. State: Replica: 40616cbd3b6a40bd860286f0be106119, State: Running, Role: LEADER I0507 06:34:03.141427 16701 consensus_queue.cc:206] T 1119750e874a4ae1b742612a6803ea81 P 40616cbd3b6a40bd860286f0be106119 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58f3e2c789b64bb0a81cc10c779b95e3" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 38173 } } peers { permanent_uuid: "40616cbd3b6a40bd860286f0be106119" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 43553 } } peers { permanent_uuid: "48ea1474af5442249334996e0babce18" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 37195 } } I0507 06:34:03.145112 16265 catalog_manager.cc:3988] T 1119750e874a4ae1b742612a6803ea81 P 40616cbd3b6a40bd860286f0be106119 reported cstate change: term changed from 0 to 1, leader changed from to 40616cbd3b6a40bd860286f0be106119 (127.0.57.193). New cstate: current_term: 1 leader_uuid: "40616cbd3b6a40bd860286f0be106119" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58f3e2c789b64bb0a81cc10c779b95e3" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 38173 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "40616cbd3b6a40bd860286f0be106119" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 43553 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "48ea1474af5442249334996e0babce18" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 37195 } health_report { overall_health: UNKNOWN } } } I0507 06:34:03.189970 231 external_mini_cluster.cc:623] 3 TS(s) registered with all masters I0507 06:34:03.191375 231 ts_itest-base.cc:232] Waiting for 1 tablets on tserver 40616cbd3b6a40bd860286f0be106119 to finish bootstrapping I0507 06:34:03.197847 231 ts_itest-base.cc:232] Waiting for 1 tablets on tserver 58f3e2c789b64bb0a81cc10c779b95e3 to finish bootstrapping I0507 06:34:03.204268 231 ts_itest-base.cc:232] Waiting for 1 tablets on tserver 48ea1474af5442249334996e0babce18 to finish bootstrapping I0507 06:34:03.225518 231 cluster_itest_util.cc:258] Not converged past 1 yet: 1.1 0.0 0.0 I0507 06:34:03.329021 231 cluster_itest_util.cc:258] Not converged past 1 yet: 1.1 0.0 0.0 I0507 06:34:03.532456 231 cluster_itest_util.cc:258] Not converged past 1 yet: 1.1 0.0 0.0 I0507 06:34:03.636284 16509 raft_consensus.cc:1181] T 1119750e874a4ae1b742612a6803ea81 P 58f3e2c789b64bb0a81cc10c779b95e3 [term 1 FOLLOWER]: Refusing update from remote peer 40616cbd3b6a40bd860286f0be106119: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch) I0507 06:34:03.637152 16701 consensus_queue.cc:984] T 1119750e874a4ae1b742612a6803ea81 P 40616cbd3b6a40bd860286f0be106119 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58f3e2c789b64bb0a81cc10c779b95e3" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 38173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0507 06:34:03.643419 16631 raft_consensus.cc:1181] T 1119750e874a4ae1b742612a6803ea81 P 48ea1474af5442249334996e0babce18 [term 1 FOLLOWER]: Refusing update from remote peer 40616cbd3b6a40bd860286f0be106119: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch) I0507 06:34:03.644134 16704 consensus_queue.cc:984] T 1119750e874a4ae1b742612a6803ea81 P 40616cbd3b6a40bd860286f0be106119 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48ea1474af5442249334996e0babce18" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 37195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0507 06:34:03.836148 231 external_mini_cluster.cc:1172] Attempting to check leaks for /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver pid 16555 I0507 06:34:03.836962 16671 generic_service.cc:171] Checking for leaks (request via RPC) I0507 06:34:03.884373 231 external_mini_cluster.cc:1131] Killing /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver with pid 16555 W0507 06:34:04.249547 16320 consensus_peers.cc:457] T 1119750e874a4ae1b742612a6803ea81 P 40616cbd3b6a40bd860286f0be106119 -> Peer 48ea1474af5442249334996e0babce18 (127.0.57.195:37195): Couldn't send request to peer 48ea1474af5442249334996e0babce18. Status: Network error: Client connection negotiation failed: client connection to 127.0.57.195:37195: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry. W0507 06:34:06.667321 16320 consensus_peers.cc:457] T 1119750e874a4ae1b742612a6803ea81 P 40616cbd3b6a40bd860286f0be106119 -> Peer 48ea1474af5442249334996e0babce18 (127.0.57.195:37195): Couldn't send request to peer 48ea1474af5442249334996e0babce18. Status: Network error: Client connection negotiation failed: client connection to 127.0.57.195:37195: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry. I0507 06:34:08.998347 16721 consensus_queue.cc:544] T 1119750e874a4ae1b742612a6803ea81 P 40616cbd3b6a40bd860286f0be106119 [LEADER]: Leader has been unable to successfully communicate with peer 48ea1474af5442249334996e0babce18 for more than 5 seconds (5.350s) W0507 06:34:09.000142 16320 consensus_peers.cc:457] T 1119750e874a4ae1b742612a6803ea81 P 40616cbd3b6a40bd860286f0be106119 -> Peer 48ea1474af5442249334996e0babce18 (127.0.57.195:37195): Couldn't send request to peer 48ea1474af5442249334996e0babce18. Status: Network error: Client connection negotiation failed: client connection to 127.0.57.195:37195: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry. I0507 06:34:09.001042 16264 catalog_manager.cc:3616] Sending ChangeConfig:ADD_PEER:NON_VOTER on tablet 1119750e874a4ae1b742612a6803ea81 (attempt 1) W0507 06:34:09.001399 16264 catalog_manager.cc:3675] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 1119750e874a4ae1b742612a6803ea81 with cas_config_opid_index -1: no extra replica candidate found for tablet 1119750e874a4ae1b742612a6803ea81 (table TestTable [id=4efac679a2544c17946c9f4b94d0b144]): Not found: could not select location for extra replica: not enough tablet servers to satisfy replica placement policy: the total number of registered tablet servers (3) does not allow for adding an extra replica; consider bringing up more to have at least 4 tablet servers up and running I0507 06:34:09.001518 16264 catalog_manager.cc:3168] Scheduling retry of ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 1119750e874a4ae1b742612a6803ea81 with cas_config_opid_index -1 with a delay of 32 ms (attempt = 1) I0507 06:34:09.034160 16247 catalog_manager.cc:3616] Sending ChangeConfig:ADD_PEER:NON_VOTER on tablet 1119750e874a4ae1b742612a6803ea81 (attempt 2) I0507 06:34:09.034468 16247 catalog_manager.cc:3168] Scheduling retry of ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 1119750e874a4ae1b742612a6803ea81 with cas_config_opid_index -1 with a delay of 80 ms (attempt = 2) I0507 06:34:09.115159 16247 catalog_manager.cc:3616] Sending ChangeConfig:ADD_PEER:NON_VOTER on tablet 1119750e874a4ae1b742612a6803ea81 (attempt 3) I0507 06:34:09.115419 16247 catalog_manager.cc:3168] Scheduling retry of ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 1119750e874a4ae1b742612a6803ea81 with cas_config_opid_index -1 with a delay of 111 ms (attempt = 3) I0507 06:34:09.227110 16247 catalog_manager.cc:3616] Sending ChangeConfig:ADD_PEER:NON_VOTER on tablet 1119750e874a4ae1b742612a6803ea81 (attempt 4) I0507 06:34:09.227401 16247 catalog_manager.cc:3168] Scheduling retry of ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 1119750e874a4ae1b742612a6803ea81 with cas_config_opid_index -1 with a delay of 150 ms (attempt = 4) I0507 06:34:09.378212 16247 catalog_manager.cc:3616] Sending ChangeConfig:ADD_PEER:NON_VOTER on tablet 1119750e874a4ae1b742612a6803ea81 (attempt 5) I0507 06:34:09.378487 16247 catalog_manager.cc:3168] Scheduling retry of ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 1119750e874a4ae1b742612a6803ea81 with cas_config_opid_index -1 with a delay of 272 ms (attempt = 5) I0507 06:34:09.651791 16247 catalog_manager.cc:3616] Sending ChangeConfig:ADD_PEER:NON_VOTER on tablet 1119750e874a4ae1b742612a6803ea81 (attempt 6) I0507 06:34:09.652062 16247 catalog_manager.cc:3168] Scheduling retry of ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 1119750e874a4ae1b742612a6803ea81 with cas_config_opid_index -1 with a delay of 548 ms (attempt = 6) I0507 06:34:10.200884 16247 catalog_manager.cc:3616] Sending ChangeConfig:ADD_PEER:NON_VOTER on tablet 1119750e874a4ae1b742612a6803ea81 (attempt 7) I0507 06:34:10.201162 16247 catalog_manager.cc:3168] Scheduling retry of ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 1119750e874a4ae1b742612a6803ea81 with cas_config_opid_index -1 with a delay of 1036 ms (attempt = 7) I0507 06:34:11.238399 16247 catalog_manager.cc:3616] Sending ChangeConfig:ADD_PEER:NON_VOTER on tablet 1119750e874a4ae1b742612a6803ea81 (attempt 8) I0507 06:34:11.238687 16247 catalog_manager.cc:3168] Scheduling retry of ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 1119750e874a4ae1b742612a6803ea81 with cas_config_opid_index -1 with a delay of 2057 ms (attempt = 8) W0507 06:34:11.724177 16320 consensus_peers.cc:457] T 1119750e874a4ae1b742612a6803ea81 P 40616cbd3b6a40bd860286f0be106119 -> Peer 48ea1474af5442249334996e0babce18 (127.0.57.195:37195): Couldn't send request to peer 48ea1474af5442249334996e0babce18. Status: Network error: Client connection negotiation failed: client connection to 127.0.57.195:37195: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry. I0507 06:34:13.296562 16247 catalog_manager.cc:3616] Sending ChangeConfig:ADD_PEER:NON_VOTER on tablet 1119750e874a4ae1b742612a6803ea81 (attempt 9) I0507 06:34:13.296887 16247 catalog_manager.cc:3168] Scheduling retry of ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 1119750e874a4ae1b742612a6803ea81 with cas_config_opid_index -1 with a delay of 4123 ms (attempt = 9) W0507 06:34:14.195538 16320 consensus_peers.cc:457] T 1119750e874a4ae1b742612a6803ea81 P 40616cbd3b6a40bd860286f0be106119 -> Peer 48ea1474af5442249334996e0babce18 (127.0.57.195:37195): Couldn't send request to peer 48ea1474af5442249334996e0babce18. Status: Network error: Client connection negotiation failed: client connection to 127.0.57.195:37195: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry. W0507 06:34:16.652081 16320 consensus_peers.cc:457] T 1119750e874a4ae1b742612a6803ea81 P 40616cbd3b6a40bd860286f0be106119 -> Peer 48ea1474af5442249334996e0babce18 (127.0.57.195:37195): Couldn't send request to peer 48ea1474af5442249334996e0babce18. Status: Network error: Client connection negotiation failed: client connection to 127.0.57.195:37195: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry. I0507 06:34:17.420940 16247 catalog_manager.cc:3616] Sending ChangeConfig:ADD_PEER:NON_VOTER on tablet 1119750e874a4ae1b742612a6803ea81 (attempt 10) I0507 06:34:17.421272 16247 catalog_manager.cc:3168] Scheduling retry of ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 1119750e874a4ae1b742612a6803ea81 with cas_config_opid_index -1 with a delay of 8192 ms (attempt = 10) I0507 06:34:18.899227 231 test_util.cc:212] Using random seed: 1746257667 I0507 06:34:18.913210 231 test_workload.cc:306] TestWorkload: Skipping table creation because table TestTable already exists I0507 06:34:18.916296 231 raft_consensus-itest-base.cc:271] Waiting until we've written at least 4MB... I0507 06:34:19.042819 16752 consensus_queue.cc:735] T 1119750e874a4ae1b742612a6803ea81 P 40616cbd3b6a40bd860286f0be106119 [LEADER]: Peer 48ea1474af5442249334996e0babce18 is lagging by at least 23 ops behind the committed index W0507 06:34:19.046135 16320 consensus_peers.cc:457] T 1119750e874a4ae1b742612a6803ea81 P 40616cbd3b6a40bd860286f0be106119 -> Peer 48ea1474af5442249334996e0babce18 (127.0.57.195:37195): Couldn't send request to peer 48ea1474af5442249334996e0babce18. Status: Network error: Client connection negotiation failed: client connection to 127.0.57.195:37195: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry. I0507 06:34:19.062772 16430 maintenance_manager.cc:306] P 40616cbd3b6a40bd860286f0be106119: Scheduling FlushMRSOp(1119750e874a4ae1b742612a6803ea81): 4601962 bytes log retention I0507 06:34:19.064365 16553 maintenance_manager.cc:306] P 58f3e2c789b64bb0a81cc10c779b95e3: Scheduling FlushMRSOp(1119750e874a4ae1b742612a6803ea81): 4207489 bytes log retention I0507 06:34:19.076411 231 raft_consensus-itest-base.cc:277] Waiting for log GC on 40616cbd3b6a40bd860286f0be106119 I0507 06:34:19.145349 16447 maintenance_manager.cc:501] P 58f3e2c789b64bb0a81cc10c779b95e3: FlushMRSOp(1119750e874a4ae1b742612a6803ea81) complete. Timing: real 0.080s user 0.063s sys 0.006s Metrics: {"bytes_written":136001,"cfile_init":1,"compiler_manager_pool.queue_time_us":3808,"data dirs.queue_time_us":691,"data dirs.run_cpu_time_us":238,"data dirs.run_wall_time_us":783,"drs_written":1,"lbm_read_time_us":77,"lbm_reads_lt_1ms":4,"lbm_write_time_us":555,"lbm_writes_lt_1ms":23,"rows_written":37,"thread_start_us":4084,"threads_started":2} I0507 06:34:19.146999 16553 maintenance_manager.cc:306] P 58f3e2c789b64bb0a81cc10c779b95e3: Scheduling LogGCOp(1119750e874a4ae1b742612a6803ea81): free 2498218 bytes of WAL I0507 06:34:19.147614 16447 log_reader.cc:377] T 1119750e874a4ae1b742612a6803ea81: removed 3 log segments from log reader I0507 06:34:19.147735 16447 log.cc:863] T 1119750e874a4ae1b742612a6803ea81 P 58f3e2c789b64bb0a81cc10c779b95e3: Deleting log segment in path: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_5.1557210500371863-231/raft_consensus-itest-cluster/ts-1/wal/wals/1119750e874a4ae1b742612a6803ea81/wal-000000001 (ops 1-8) I0507 06:34:19.147855 16447 log.cc:863] T 1119750e874a4ae1b742612a6803ea81 P 58f3e2c789b64bb0a81cc10c779b95e3: Deleting log segment in path: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_5.1557210500371863-231/raft_consensus-itest-cluster/ts-1/wal/wals/1119750e874a4ae1b742612a6803ea81/wal-000000002 (ops 9-14) I0507 06:34:19.147922 16447 log.cc:863] T 1119750e874a4ae1b742612a6803ea81 P 58f3e2c789b64bb0a81cc10c779b95e3: Deleting log segment in path: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_5.1557210500371863-231/raft_consensus-itest-cluster/ts-1/wal/wals/1119750e874a4ae1b742612a6803ea81/wal-000000003 (ops 15-20) I0507 06:34:19.148574 16447 maintenance_manager.cc:501] P 58f3e2c789b64bb0a81cc10c779b95e3: LogGCOp(1119750e874a4ae1b742612a6803ea81) complete. Timing: real 0.001s user 0.000s sys 0.001s Metrics: {} I0507 06:34:19.144065 16324 maintenance_manager.cc:501] P 40616cbd3b6a40bd860286f0be106119: FlushMRSOp(1119750e874a4ae1b742612a6803ea81) complete. Timing: real 0.081s user 0.052s sys 0.016s Metrics: {"bytes_written":136001,"cfile_init":1,"compiler_manager_pool.queue_time_us":8371,"data dirs.queue_time_us":647,"data dirs.run_cpu_time_us":226,"data dirs.run_wall_time_us":778,"drs_written":1,"lbm_read_time_us":71,"lbm_reads_lt_1ms":4,"lbm_write_time_us":574,"lbm_writes_lt_1ms":23,"rows_written":36,"thread_start_us":13146,"threads_started":2} I0507 06:34:19.148970 16553 maintenance_manager.cc:306] P 58f3e2c789b64bb0a81cc10c779b95e3: Scheduling UndoDeltaBlockGCOp(1119750e874a4ae1b742612a6803ea81): 551 bytes on disk I0507 06:34:19.149678 16430 maintenance_manager.cc:306] P 40616cbd3b6a40bd860286f0be106119: Scheduling LogGCOp(1119750e874a4ae1b742612a6803ea81): free 2761184 bytes of WAL I0507 06:34:19.150007 16324 log_reader.cc:377] T 1119750e874a4ae1b742612a6803ea81: removed 3 log segments from log reader I0507 06:34:19.150080 16324 log.cc:863] T 1119750e874a4ae1b742612a6803ea81 P 40616cbd3b6a40bd860286f0be106119: Deleting log segment in path: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_5.1557210500371863-231/raft_consensus-itest-cluster/ts-0/wal/wals/1119750e874a4ae1b742612a6803ea81/wal-000000001 (ops 1-8) I0507 06:34:19.150187 16324 log.cc:863] T 1119750e874a4ae1b742612a6803ea81 P 40616cbd3b6a40bd860286f0be106119: Deleting log segment in path: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_5.1557210500371863-231/raft_consensus-itest-cluster/ts-0/wal/wals/1119750e874a4ae1b742612a6803ea81/wal-000000002 (ops 9-15) I0507 06:34:19.150246 16324 log.cc:863] T 1119750e874a4ae1b742612a6803ea81 P 40616cbd3b6a40bd860286f0be106119: Deleting log segment in path: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_5.1557210500371863-231/raft_consensus-itest-cluster/ts-0/wal/wals/1119750e874a4ae1b742612a6803ea81/wal-000000003 (ops 16-22) I0507 06:34:19.150880 16324 maintenance_manager.cc:501] P 40616cbd3b6a40bd860286f0be106119: LogGCOp(1119750e874a4ae1b742612a6803ea81) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {} I0507 06:34:19.151337 16430 maintenance_manager.cc:306] P 40616cbd3b6a40bd860286f0be106119: Scheduling UndoDeltaBlockGCOp(1119750e874a4ae1b742612a6803ea81): 536 bytes on disk I0507 06:34:19.151715 16447 maintenance_manager.cc:501] P 58f3e2c789b64bb0a81cc10c779b95e3: UndoDeltaBlockGCOp(1119750e874a4ae1b742612a6803ea81) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":70,"lbm_reads_lt_1ms":4} I0507 06:34:19.152809 16324 maintenance_manager.cc:501] P 40616cbd3b6a40bd860286f0be106119: UndoDeltaBlockGCOp(1119750e874a4ae1b742612a6803ea81) complete. Timing: real 0.001s user 0.001s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":110,"lbm_reads_lt_1ms":4} I0507 06:34:19.158767 231 raft_consensus-itest-base.cc:282] Log GC complete on 40616cbd3b6a40bd860286f0be106119 I0507 06:34:19.624263 16769 consensus_queue.cc:689] T 1119750e874a4ae1b742612a6803ea81 P 40616cbd3b6a40bd860286f0be106119 [LEADER]: The logs necessary to catch up peer 48ea1474af5442249334996e0babce18 have been garbage collected. The follower will never be able to catch up (Not found: Failed to read ops 2..32: Segment 1 which contained index 2 has been GCed) I0507 06:34:19.624359 16769 consensus_queue.cc:544] T 1119750e874a4ae1b742612a6803ea81 P 40616cbd3b6a40bd860286f0be106119 [LEADER]: The logs necessary to catch up peer 48ea1474af5442249334996e0babce18 have been garbage collected. The replica will never be able to catch up I0507 06:34:19.627216 16262 catalog_manager.cc:3739] Sending ChangeConfig:REMOVE_PEER on tablet 1119750e874a4ae1b742612a6803ea81 (attempt 1) I0507 06:34:19.629561 16386 consensus_queue.cc:206] T 1119750e874a4ae1b742612a6803ea81 P 40616cbd3b6a40bd860286f0be106119 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 39, Committed index: 39, Last appended: 1.39, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 40 OBSOLETE_local: false peers { permanent_uuid: "58f3e2c789b64bb0a81cc10c779b95e3" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 38173 } } peers { permanent_uuid: "40616cbd3b6a40bd860286f0be106119" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 43553 } } I0507 06:34:19.631456 16509 raft_consensus.cc:1181] T 1119750e874a4ae1b742612a6803ea81 P 58f3e2c789b64bb0a81cc10c779b95e3 [term 1 FOLLOWER]: Refusing update from remote peer 40616cbd3b6a40bd860286f0be106119: Log matching property violated. Preceding OpId in replica: term: 1 index: 39. Preceding OpId from leader: term: 1 index: 40. (index mismatch) I0507 06:34:19.632027 16769 consensus_queue.cc:984] T 1119750e874a4ae1b742612a6803ea81 P 40616cbd3b6a40bd860286f0be106119 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58f3e2c789b64bb0a81cc10c779b95e3" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 38173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 40, Last known committed idx: 39, Time since last communication: 0.000s I0507 06:34:19.634430 16770 raft_consensus.cc:2788] T 1119750e874a4ae1b742612a6803ea81 P 40616cbd3b6a40bd860286f0be106119 [term 1 LEADER]: Committing config change with OpId 1.40: config changed from index -1 to 40, VOTER 48ea1474af5442249334996e0babce18 (127.0.57.195) evicted. New config: { opid_index: 40 OBSOLETE_local: false peers { permanent_uuid: "58f3e2c789b64bb0a81cc10c779b95e3" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 38173 } } peers { permanent_uuid: "40616cbd3b6a40bd860286f0be106119" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 43553 } } } I0507 06:34:19.634958 16509 raft_consensus.cc:2788] T 1119750e874a4ae1b742612a6803ea81 P 58f3e2c789b64bb0a81cc10c779b95e3 [term 1 FOLLOWER]: Committing config change with OpId 1.40: config changed from index -1 to 40, VOTER 48ea1474af5442249334996e0babce18 (127.0.57.195) evicted. New config: { opid_index: 40 OBSOLETE_local: false peers { permanent_uuid: "58f3e2c789b64bb0a81cc10c779b95e3" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 38173 } } peers { permanent_uuid: "40616cbd3b6a40bd860286f0be106119" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 43553 } } } I0507 06:34:19.637336 16249 catalog_manager.cc:3531] ChangeConfig:REMOVE_PEER RPC for tablet 1119750e874a4ae1b742612a6803ea81 with cas_config_opid_index -1: ChangeConfig:REMOVE_PEER succeeded (attempt 1) I0507 06:34:19.638268 16262 catalog_manager.cc:3988] T 1119750e874a4ae1b742612a6803ea81 P 40616cbd3b6a40bd860286f0be106119 reported cstate change: config changed from index -1 to 40, VOTER 48ea1474af5442249334996e0babce18 (127.0.57.195) evicted. New cstate: current_term: 1 leader_uuid: "40616cbd3b6a40bd860286f0be106119" committed_config { opid_index: 40 OBSOLETE_local: false peers { permanent_uuid: "58f3e2c789b64bb0a81cc10c779b95e3" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 38173 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "40616cbd3b6a40bd860286f0be106119" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 43553 } health_report { overall_health: HEALTHY } } } I0507 06:34:19.642530 16262 catalog_manager.cc:3390] Sending DeleteTablet(TABLET_DATA_TOMBSTONED) for tablet 1119750e874a4ae1b742612a6803ea81 on 48ea1474af5442249334996e0babce18 (127.0.57.195:37195) (TS 48ea1474af5442249334996e0babce18 not found in new config with opid_index 40) I0507 06:34:19.642832 16262 catalog_manager.cc:3616] Sending ChangeConfig:ADD_PEER:NON_VOTER on tablet 1119750e874a4ae1b742612a6803ea81 (attempt 1) I0507 06:34:19.644742 16262 catalog_manager.cc:3616] Sending ChangeConfig:ADD_PEER:NON_VOTER on tablet 1119750e874a4ae1b742612a6803ea81 (attempt 1) W0507 06:34:19.649248 16249 catalog_manager.cc:3132] TS 48ea1474af5442249334996e0babce18 (127.0.57.195:37195): Delete Tablet RPC failed for tablet 1119750e874a4ae1b742612a6803ea81: Network error: Client connection negotiation failed: client connection to 127.0.57.195:37195: connect: Connection refused (error 111) I0507 06:34:19.649389 16249 catalog_manager.cc:3168] Scheduling retry of 1119750e874a4ae1b742612a6803ea81 Delete Tablet RPC for TS=48ea1474af5442249334996e0babce18 with a delay of 19 ms (attempt = 1) I0507 06:34:19.649555 16386 consensus_queue.cc:206] T 1119750e874a4ae1b742612a6803ea81 P 40616cbd3b6a40bd860286f0be106119 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 40, Committed index: 40, Last appended: 1.40, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 41 OBSOLETE_local: false peers { permanent_uuid: "58f3e2c789b64bb0a81cc10c779b95e3" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 38173 } } peers { permanent_uuid: "40616cbd3b6a40bd860286f0be106119" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 43553 } } peers { permanent_uuid: "48ea1474af5442249334996e0babce18" member_type: NON_VOTER last_known_addr { host: "127.0.57.195" port: 37195 } attrs { promote: true } } I0507 06:34:19.651438 16509 raft_consensus.cc:1181] T 1119750e874a4ae1b742612a6803ea81 P 58f3e2c789b64bb0a81cc10c779b95e3 [term 1 FOLLOWER]: Refusing update from remote peer 40616cbd3b6a40bd860286f0be106119: Log matching property violated. Preceding OpId in replica: term: 1 index: 40. Preceding OpId from leader: term: 1 index: 41. (index mismatch) I0507 06:34:19.652027 16769 consensus_queue.cc:984] T 1119750e874a4ae1b742612a6803ea81 P 40616cbd3b6a40bd860286f0be106119 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58f3e2c789b64bb0a81cc10c779b95e3" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 38173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 41, Last known committed idx: 40, Time since last communication: 0.000s I0507 06:34:19.654608 16770 raft_consensus.cc:2788] T 1119750e874a4ae1b742612a6803ea81 P 40616cbd3b6a40bd860286f0be106119 [term 1 LEADER]: Committing config change with OpId 1.41: config changed from index 40 to 41, NON_VOTER 48ea1474af5442249334996e0babce18 (127.0.57.195) added. New config: { opid_index: 41 OBSOLETE_local: false peers { permanent_uuid: "58f3e2c789b64bb0a81cc10c779b95e3" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 38173 } } peers { permanent_uuid: "40616cbd3b6a40bd860286f0be106119" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 43553 } } peers { permanent_uuid: "48ea1474af5442249334996e0babce18" member_type: NON_VOTER last_known_addr { host: "127.0.57.195" port: 37195 } attrs { promote: true } } } W0507 06:34:19.657793 16320 consensus_peers.cc:457] T 1119750e874a4ae1b742612a6803ea81 P 40616cbd3b6a40bd860286f0be106119 -> Peer 48ea1474af5442249334996e0babce18 (127.0.57.195:37195): Couldn't send request to peer 48ea1474af5442249334996e0babce18. Status: Network error: Client connection negotiation failed: client connection to 127.0.57.195:37195: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry. I0507 06:34:19.658311 16249 catalog_manager.cc:3548] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 1119750e874a4ae1b742612a6803ea81 with cas_config_opid_index 40: ChangeConfig:ADD_PEER:NON_VOTER failed with leader 40616cbd3b6a40bd860286f0be106119 (127.0.57.193:43553) due to error UNKNOWN_ERROR; will retry: Illegal state: RaftConfig change currently pending. Only one is allowed at a time. Committed config: opid_index: 40 OBSOLETE_local: false peers { permanent_uuid: "58f3e2c789b64bb0a81cc10c779b95e3" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 38173 } } peers { permanent_uuid: "40616cbd3b6a40bd860286f0be106119" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 43553 } }. Pending config: opid_index: 41 OBSOLETE_local: false peers { permanent_uuid: "58f3e2c789b64bb0a81cc10c779b95e3" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 38173 } } peers { permanent_uuid: "40616cbd3b6a40bd860286f0be106119" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 43553 } } peers { permanent_uuid: "48ea1474af5442249334996e0babce18" member_type: NON_VOTER last_known_addr { host: "127.0.57.195" port: 37195 } attrs { promote: true } } I0507 06:34:19.658424 16249 catalog_manager.cc:3168] Scheduling retry of ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 1119750e874a4ae1b742612a6803ea81 with cas_config_opid_index 40 with a delay of 63 ms (attempt = 1) I0507 06:34:19.658576 16249 catalog_manager.cc:3531] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 1119750e874a4ae1b742612a6803ea81 with cas_config_opid_index 40: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1) I0507 06:34:19.658782 16262 catalog_manager.cc:3988] T 1119750e874a4ae1b742612a6803ea81 P 40616cbd3b6a40bd860286f0be106119 reported cstate change: config changed from index 40 to 41, NON_VOTER 48ea1474af5442249334996e0babce18 (127.0.57.195) added. New cstate: current_term: 1 leader_uuid: "40616cbd3b6a40bd860286f0be106119" committed_config { opid_index: 41 OBSOLETE_local: false peers { permanent_uuid: "58f3e2c789b64bb0a81cc10c779b95e3" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 38173 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "40616cbd3b6a40bd860286f0be106119" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 43553 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "48ea1474af5442249334996e0babce18" member_type: NON_VOTER last_known_addr { host: "127.0.57.195" port: 37195 } attrs { promote: true } health_report { overall_health: UNKNOWN } } } I0507 06:34:19.659938 16509 raft_consensus.cc:2788] T 1119750e874a4ae1b742612a6803ea81 P 58f3e2c789b64bb0a81cc10c779b95e3 [term 1 FOLLOWER]: Committing config change with OpId 1.41: config changed from index 40 to 41, NON_VOTER 48ea1474af5442249334996e0babce18 (127.0.57.195) added. New config: { opid_index: 41 OBSOLETE_local: false peers { permanent_uuid: "58f3e2c789b64bb0a81cc10c779b95e3" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 38173 } } peers { permanent_uuid: "40616cbd3b6a40bd860286f0be106119" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 43553 } } peers { permanent_uuid: "48ea1474af5442249334996e0babce18" member_type: NON_VOTER last_known_addr { host: "127.0.57.195" port: 37195 } attrs { promote: true } } } I0507 06:34:19.668979 16249 catalog_manager.cc:3390] Sending DeleteTablet(TABLET_DATA_TOMBSTONED) for tablet 1119750e874a4ae1b742612a6803ea81 on 48ea1474af5442249334996e0babce18 (127.0.57.195:37195) (TS 48ea1474af5442249334996e0babce18 not found in new config with opid_index 40) W0507 06:34:19.669783 16249 catalog_manager.cc:3132] TS 48ea1474af5442249334996e0babce18 (127.0.57.195:37195): Delete Tablet RPC failed for tablet 1119750e874a4ae1b742612a6803ea81: Network error: Client connection negotiation failed: client connection to 127.0.57.195:37195: connect: Connection refused (error 111) I0507 06:34:19.669872 16249 catalog_manager.cc:3168] Scheduling retry of 1119750e874a4ae1b742612a6803ea81 Delete Tablet RPC for TS=48ea1474af5442249334996e0babce18 with a delay of 64 ms (attempt = 2) I0507 06:34:19.722563 16249 catalog_manager.cc:3564] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 1119750e874a4ae1b742612a6803ea81 with cas_config_opid_index 40: aborting the task: latest config opid_index 41; task opid_index 40 I0507 06:34:19.735110 16249 catalog_manager.cc:3390] Sending DeleteTablet(TABLET_DATA_TOMBSTONED) for tablet 1119750e874a4ae1b742612a6803ea81 on 48ea1474af5442249334996e0babce18 (127.0.57.195:37195) (TS 48ea1474af5442249334996e0babce18 not found in new config with opid_index 40) W0507 06:34:19.736165 16249 catalog_manager.cc:3132] TS 48ea1474af5442249334996e0babce18 (127.0.57.195:37195): Delete Tablet RPC failed for tablet 1119750e874a4ae1b742612a6803ea81: Network error: Client connection negotiation failed: client connection to 127.0.57.195:37195: connect: Connection refused (error 111) I0507 06:34:19.736263 16249 catalog_manager.cc:3168] Scheduling retry of 1119750e874a4ae1b742612a6803ea81 Delete Tablet RPC for TS=48ea1474af5442249334996e0babce18 with a delay of 67 ms (attempt = 3) I0507 06:34:19.803933 16249 catalog_manager.cc:3390] Sending DeleteTablet(TABLET_DATA_TOMBSTONED) for tablet 1119750e874a4ae1b742612a6803ea81 on 48ea1474af5442249334996e0babce18 (127.0.57.195:37195) (TS 48ea1474af5442249334996e0babce18 not found in new config with opid_index 40) W0507 06:34:19.805027 16249 catalog_manager.cc:3132] TS 48ea1474af5442249334996e0babce18 (127.0.57.195:37195): Delete Tablet RPC failed for tablet 1119750e874a4ae1b742612a6803ea81: Network error: Client connection negotiation failed: client connection to 127.0.57.195:37195: connect: Connection refused (error 111) I0507 06:34:19.805104 16249 catalog_manager.cc:3168] Scheduling retry of 1119750e874a4ae1b742612a6803ea81 Delete Tablet RPC for TS=48ea1474af5442249334996e0babce18 with a delay of 163 ms (attempt = 4) I0507 06:34:19.969063 16249 catalog_manager.cc:3390] Sending DeleteTablet(TABLET_DATA_TOMBSTONED) for tablet 1119750e874a4ae1b742612a6803ea81 on 48ea1474af5442249334996e0babce18 (127.0.57.195:37195) (TS 48ea1474af5442249334996e0babce18 not found in new config with opid_index 40) W0507 06:34:19.970175 16249 catalog_manager.cc:3132] TS 48ea1474af5442249334996e0babce18 (127.0.57.195:37195): Delete Tablet RPC failed for tablet 1119750e874a4ae1b742612a6803ea81: Network error: Client connection negotiation failed: client connection to 127.0.57.195:37195: connect: Connection refused (error 111) I0507 06:34:19.970273 16249 catalog_manager.cc:3168] Scheduling retry of 1119750e874a4ae1b742612a6803ea81 Delete Tablet RPC for TS=48ea1474af5442249334996e0babce18 with a delay of 257 ms (attempt = 5) I0507 06:34:20.228633 16249 catalog_manager.cc:3390] Sending DeleteTablet(TABLET_DATA_TOMBSTONED) for tablet 1119750e874a4ae1b742612a6803ea81 on 48ea1474af5442249334996e0babce18 (127.0.57.195:37195) (TS 48ea1474af5442249334996e0babce18 not found in new config with opid_index 40) W0507 06:34:20.229668 16249 catalog_manager.cc:3132] TS 48ea1474af5442249334996e0babce18 (127.0.57.195:37195): Delete Tablet RPC failed for tablet 1119750e874a4ae1b742612a6803ea81: Network error: Client connection negotiation failed: client connection to 127.0.57.195:37195: connect: Connection refused (error 111) I0507 06:34:20.229753 16249 catalog_manager.cc:3168] Scheduling retry of 1119750e874a4ae1b742612a6803ea81 Delete Tablet RPC for TS=48ea1474af5442249334996e0babce18 with a delay of 519 ms (attempt = 6) I0507 06:34:20.749701 16249 catalog_manager.cc:3390] Sending DeleteTablet(TABLET_DATA_TOMBSTONED) for tablet 1119750e874a4ae1b742612a6803ea81 on 48ea1474af5442249334996e0babce18 (127.0.57.195:37195) (TS 48ea1474af5442249334996e0babce18 not found in new config with opid_index 40) W0507 06:34:20.751111 16249 catalog_manager.cc:3132] TS 48ea1474af5442249334996e0babce18 (127.0.57.195:37195): Delete Tablet RPC failed for tablet 1119750e874a4ae1b742612a6803ea81: Network error: Client connection negotiation failed: client connection to 127.0.57.195:37195: connect: Connection refused (error 111) I0507 06:34:20.751196 16249 catalog_manager.cc:3168] Scheduling retry of 1119750e874a4ae1b742612a6803ea81 Delete Tablet RPC for TS=48ea1474af5442249334996e0babce18 with a delay of 1072 ms (attempt = 7) I0507 06:34:21.160804 231 raft_consensus-itest-base.cc:298] Servers converged with original term 1 I0507 06:34:21.167363 231 external_mini_cluster.cc:933] Running /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_5.1557210500371863-231/raft_consensus-itest-cluster/ts-2/wal --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_5.1557210500371863-231/raft_consensus-itest-cluster/ts-2/data --block_manager=log --rpc_bind_addresses=127.0.57.195:37195 --local_ip_for_outbound_sockets=127.0.57.195 --webserver_port=36047 --webserver_interface=127.0.57.195 --tserver_master_addrs=127.0.57.254:38831 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_5.1557210500371863-231/raft_consensus-itest-cluster/ts-2/logs --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_5.1557210500371863-231/raft_consensus-itest-cluster/ts-2/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=true --follower_unavailable_considered_failed_sec=5 --log_compression_codec=none --log_cache_size_limit_mb=1 --log_segment_size_mb=1 --log_async_preallocate_segments=false --log_min_segments_to_retain=1 --log_max_segments_to_retain=3 --maintenance_manager_polling_interval_ms=100 --log_target_replay_size_mb=1 --max_cell_size_bytes=1000000 with env {} WARNING: Logging before InitGoogleLogging() is written to STDERR W0507 06:34:21.354450 16781 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0507 06:34:21.354713 16781 flags.cc:404] Enabled unsafe flag: --max_cell_size_bytes=1000000 W0507 06:34:21.354784 16781 flags.cc:404] Enabled unsafe flag: --never_fsync=true W0507 06:34:21.356990 16781 flags.cc:404] Enabled experimental flag: --log_compression_codec=none W0507 06:34:21.357053 16781 flags.cc:404] Enabled experimental flag: --log_max_segments_to_retain=3 W0507 06:34:21.357118 16781 flags.cc:404] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true W0507 06:34:21.357187 16781 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W0507 06:34:21.357312 16781 flags.cc:404] Enabled experimental flag: --log_target_replay_size_mb=1 W0507 06:34:21.357388 16781 flags.cc:404] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.57.195 I0507 06:34:21.361901 16781 tablet_server_main.cc:78] Tablet server non-default flags: --follower_unavailable_considered_failed_sec=5 --log_compression_codec=none --log_max_segments_to_retain=3 --log_cache_size_limit_mb=1 --log_async_preallocate_segments=false --log_segment_size_mb=1 --fs_data_dirs=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_5.1557210500371863-231/raft_consensus-itest-cluster/ts-2/data --fs_wal_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_5.1557210500371863-231/raft_consensus-itest-cluster/ts-2/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.0.57.195:37195 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_5.1557210500371863-231/raft_consensus-itest-cluster/ts-2/data/info.pb --webserver_interface=127.0.57.195 --webserver_port=36047 --max_cell_size_bytes=1000000 --tserver_master_addrs=127.0.57.254:38831 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --log_target_replay_size_mb=1 --maintenance_manager_polling_interval_ms=100 --enable_minidumps=false --local_ip_for_outbound_sockets=127.0.57.195 --log_dir=/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_5.1557210500371863-231/raft_consensus-itest-cluster/ts-2/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.10.0-cdh6.x-SNAPSHOT revision 7d0ee26697926e87af9ef395008799577f6ab43f build type FASTDEBUG built by jenkins at 06 May 2019 22:44:21 PST on quasar-knmpfx-1.vpc.cloudera.com build id 322017 ASAN enabled I0507 06:34:21.370302 16781 system_ntp.cc:202] NTP initialized. Skew: 500ppm Current error: 205016us I0507 06:34:21.371131 16781 fs_manager.cc:263] Metadata directory not provided I0507 06:34:21.371229 16781 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0507 06:34:21.377696 16781 fs_manager.cc:399] Time spent opening directory manager: real 0.005s user 0.002s sys 0.001s I0507 06:34:21.378499 16781 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0507 06:34:21.379170 16781 file_cache.cc:466] Constructed file cache lbm with capacity 419430 I0507 06:34:21.382652 16781 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s I0507 06:34:21.382845 16781 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_5.1557210500371863-231/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_5.1557210500371863-231/raft_consensus-itest-cluster/ts-2/wal uuid: "48ea1474af5442249334996e0babce18" format_stamp: "Formatted at 2019-05-07 06:34:02 on dist-test-slave-dist-test-slave-rqp6" I0507 06:34:21.383981 16781 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_5.1557210500371863-231/raft_consensus-itest-cluster/ts-2/wal metadata directory: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_5.1557210500371863-231/raft_consensus-itest-cluster/ts-2/wal 1 data directories: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_5.1557210500371863-231/raft_consensus-itest-cluster/ts-2/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0507 06:34:21.420163 16781 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0507 06:34:21.422310 16781 ts_tablet_manager.cc:344] Loading tablet metadata (0/1 complete) I0507 06:34:21.426744 16781 ts_tablet_manager.cc:356] Loaded tablet metadata (1 total tablets, 1 live tablets) I0507 06:34:21.426827 16781 ts_tablet_manager.cc:362] Registering tablets (0/1 complete) I0507 06:34:21.430002 16781 ts_tablet_manager.cc:376] Registered 1 tablets I0507 06:34:21.430793 16796 tablet_bootstrap.cc:439] T 1119750e874a4ae1b742612a6803ea81 P 48ea1474af5442249334996e0babce18: Bootstrap starting. I0507 06:34:21.444813 16796 log.cc:525] T 1119750e874a4ae1b742612a6803ea81 P 48ea1474af5442249334996e0babce18: Log is configured to *not* fsync() on all Append() calls I0507 06:34:21.449373 16796 tablet_bootstrap.cc:439] T 1119750e874a4ae1b742612a6803ea81 P 48ea1474af5442249334996e0babce18: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I0507 06:34:21.450201 16796 tablet_bootstrap.cc:439] T 1119750e874a4ae1b742612a6803ea81 P 48ea1474af5442249334996e0babce18: Bootstrap complete. I0507 06:34:21.450858 16796 ts_tablet_manager.cc:1053] T 1119750e874a4ae1b742612a6803ea81 P 48ea1474af5442249334996e0babce18: Time spent bootstrapping tablet: real 0.020s user 0.014s sys 0.003s I0507 06:34:21.464277 16796 raft_consensus.cc:340] T 1119750e874a4ae1b742612a6803ea81 P 48ea1474af5442249334996e0babce18 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58f3e2c789b64bb0a81cc10c779b95e3" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 38173 } } peers { permanent_uuid: "40616cbd3b6a40bd860286f0be106119" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 43553 } } peers { permanent_uuid: "48ea1474af5442249334996e0babce18" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 37195 } } I0507 06:34:21.469506 16796 raft_consensus.cc:705] T 1119750e874a4ae1b742612a6803ea81 P 48ea1474af5442249334996e0babce18 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48ea1474af5442249334996e0babce18, State: Initialized, Role: FOLLOWER I0507 06:34:21.469527 16781 rpc_server.cc:205] RPC server started. Bound to: 127.0.57.195:37195 I0507 06:34:21.470260 16796 consensus_queue.cc:229] T 1119750e874a4ae1b742612a6803ea81 P 48ea1474af5442249334996e0babce18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58f3e2c789b64bb0a81cc10c779b95e3" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 38173 } } peers { permanent_uuid: "40616cbd3b6a40bd860286f0be106119" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 43553 } } peers { permanent_uuid: "48ea1474af5442249334996e0babce18" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 37195 } } I0507 06:34:21.471276 16781 webserver.cc:308] Webserver started at http://127.0.57.195:36047/ using document root and password file I0507 06:34:21.471909 16796 ts_tablet_manager.cc:1081] T 1119750e874a4ae1b742612a6803ea81 P 48ea1474af5442249334996e0babce18: Time spent starting tablet: real 0.021s user 0.013s sys 0.005s I0507 06:34:21.472167 16781 server_base.cc:608] Dumped server information to /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_5.1557210500371863-231/raft_consensus-itest-cluster/ts-2/data/info.pb I0507 06:34:21.475066 231 external_mini_cluster.cc:995] Started /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver as pid 16781 I0507 06:34:21.481525 16903 heartbeater.cc:345] Connected to a master server at 127.0.57.254:38831 I0507 06:34:21.481767 16903 heartbeater.cc:426] Registering TS with master... I0507 06:34:21.482398 16903 heartbeater.cc:475] Master 127.0.57.254:38831 requested a full tablet report, sending... I0507 06:34:21.484130 16263 ts_manager.cc:159] Re-registered known tserver with Master: 48ea1474af5442249334996e0babce18 (127.0.57.195:37195) I0507 06:34:21.485340 16263 master_service.cc:273] Signed X509 certificate for tserver {username='slave'} at 127.0.57.195:37155 I0507 06:34:21.616161 16859 raft_consensus.cc:1181] T 1119750e874a4ae1b742612a6803ea81 P 48ea1474af5442249334996e0babce18 [term 1 FOLLOWER]: Refusing update from remote peer 40616cbd3b6a40bd860286f0be106119: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 40. (index mismatch) I0507 06:34:21.617650 16769 consensus_queue.cc:986] T 1119750e874a4ae1b742612a6803ea81 P 40616cbd3b6a40bd860286f0be106119 [LEADER]: Got LMP mismatch error from peer: Peer: permanent_uuid: "48ea1474af5442249334996e0babce18" member_type: NON_VOTER last_known_addr { host: "127.0.57.195" port: 37195 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 41, Last known committed idx: 1, Time since last communication: 0.000s I0507 06:34:21.617995 16769 consensus_queue.cc:689] T 1119750e874a4ae1b742612a6803ea81 P 40616cbd3b6a40bd860286f0be106119 [LEADER]: The logs necessary to catch up peer 48ea1474af5442249334996e0babce18 have been garbage collected. The follower will never be able to catch up (Not found: Failed to read ops 2..40: Segment 1 which contained index 2 has been GCed) I0507 06:34:21.618072 16769 consensus_queue.cc:544] T 1119750e874a4ae1b742612a6803ea81 P 40616cbd3b6a40bd860286f0be106119 [LEADER]: The logs necessary to catch up peer 48ea1474af5442249334996e0babce18 have been garbage collected. The replica will never be able to catch up I0507 06:34:21.620373 16263 catalog_manager.cc:3739] Sending ChangeConfig:REMOVE_PEER on tablet 1119750e874a4ae1b742612a6803ea81 (attempt 1) I0507 06:34:21.621625 16385 consensus_queue.cc:206] T 1119750e874a4ae1b742612a6803ea81 P 40616cbd3b6a40bd860286f0be106119 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 41, Committed index: 41, Last appended: 1.41, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 42 OBSOLETE_local: false peers { permanent_uuid: "58f3e2c789b64bb0a81cc10c779b95e3" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 38173 } } peers { permanent_uuid: "40616cbd3b6a40bd860286f0be106119" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 43553 } } I0507 06:34:21.628695 16509 raft_consensus.cc:1181] T 1119750e874a4ae1b742612a6803ea81 P 58f3e2c789b64bb0a81cc10c779b95e3 [term 1 FOLLOWER]: Refusing update from remote peer 40616cbd3b6a40bd860286f0be106119: Log matching property violated. Preceding OpId in replica: term: 1 index: 41. Preceding OpId from leader: term: 1 index: 42. (index mismatch) I0507 06:34:21.629354 16909 consensus_queue.cc:984] T 1119750e874a4ae1b742612a6803ea81 P 40616cbd3b6a40bd860286f0be106119 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58f3e2c789b64bb0a81cc10c779b95e3" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 38173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 42, Last known committed idx: 41, Time since last communication: 0.000s I0507 06:34:21.631793 16769 raft_consensus.cc:2788] T 1119750e874a4ae1b742612a6803ea81 P 40616cbd3b6a40bd860286f0be106119 [term 1 LEADER]: Committing config change with OpId 1.42: config changed from index 41 to 42, NON_VOTER 48ea1474af5442249334996e0babce18 (127.0.57.195) evicted. New config: { opid_index: 42 OBSOLETE_local: false peers { permanent_uuid: "58f3e2c789b64bb0a81cc10c779b95e3" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 38173 } } peers { permanent_uuid: "40616cbd3b6a40bd860286f0be106119" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 43553 } } } I0507 06:34:21.632262 16509 raft_consensus.cc:2788] T 1119750e874a4ae1b742612a6803ea81 P 58f3e2c789b64bb0a81cc10c779b95e3 [term 1 FOLLOWER]: Committing config change with OpId 1.42: config changed from index 41 to 42, NON_VOTER 48ea1474af5442249334996e0babce18 (127.0.57.195) evicted. New config: { opid_index: 42 OBSOLETE_local: false peers { permanent_uuid: "58f3e2c789b64bb0a81cc10c779b95e3" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 38173 } } peers { permanent_uuid: "40616cbd3b6a40bd860286f0be106119" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 43553 } } } I0507 06:34:21.634485 16249 catalog_manager.cc:3531] ChangeConfig:REMOVE_PEER RPC for tablet 1119750e874a4ae1b742612a6803ea81 with cas_config_opid_index 41: ChangeConfig:REMOVE_PEER succeeded (attempt 1) I0507 06:34:21.635490 16263 catalog_manager.cc:3988] T 1119750e874a4ae1b742612a6803ea81 P 40616cbd3b6a40bd860286f0be106119 reported cstate change: config changed from index 41 to 42, NON_VOTER 48ea1474af5442249334996e0babce18 (127.0.57.195) evicted. New cstate: current_term: 1 leader_uuid: "40616cbd3b6a40bd860286f0be106119" committed_config { opid_index: 42 OBSOLETE_local: false peers { permanent_uuid: "58f3e2c789b64bb0a81cc10c779b95e3" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 38173 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "40616cbd3b6a40bd860286f0be106119" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 43553 } health_report { overall_health: HEALTHY } } } I0507 06:34:21.641019 16263 catalog_manager.cc:3390] Sending DeleteTablet(TABLET_DATA_TOMBSTONED) for tablet 1119750e874a4ae1b742612a6803ea81 on 48ea1474af5442249334996e0babce18 (127.0.57.195:37195) (TS 48ea1474af5442249334996e0babce18 not found in new config with opid_index 42) I0507 06:34:21.641207 16263 catalog_manager.cc:3616] Sending ChangeConfig:ADD_PEER:NON_VOTER on tablet 1119750e874a4ae1b742612a6803ea81 (attempt 1) I0507 06:34:21.642854 16385 consensus_queue.cc:206] T 1119750e874a4ae1b742612a6803ea81 P 40616cbd3b6a40bd860286f0be106119 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 42, Committed index: 42, Last appended: 1.42, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 43 OBSOLETE_local: false peers { permanent_uuid: "58f3e2c789b64bb0a81cc10c779b95e3" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 38173 } } peers { permanent_uuid: "40616cbd3b6a40bd860286f0be106119" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 43553 } } peers { permanent_uuid: "48ea1474af5442249334996e0babce18" member_type: NON_VOTER last_known_addr { host: "127.0.57.195" port: 37195 } attrs { promote: true } } I0507 06:34:21.644712 16509 raft_consensus.cc:1181] T 1119750e874a4ae1b742612a6803ea81 P 58f3e2c789b64bb0a81cc10c779b95e3 [term 1 FOLLOWER]: Refusing update from remote peer 40616cbd3b6a40bd860286f0be106119: Log matching property violated. Preceding OpId in replica: term: 1 index: 42. Preceding OpId from leader: term: 1 index: 43. (index mismatch) I0507 06:34:21.645362 16769 consensus_queue.cc:984] T 1119750e874a4ae1b742612a6803ea81 P 40616cbd3b6a40bd860286f0be106119 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58f3e2c789b64bb0a81cc10c779b95e3" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 38173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 43, Last known committed idx: 42, Time since last communication: 0.000s I0507 06:34:21.645606 16859 raft_consensus.cc:1181] T 1119750e874a4ae1b742612a6803ea81 P 48ea1474af5442249334996e0babce18 [term 1 FOLLOWER]: Refusing update from remote peer 40616cbd3b6a40bd860286f0be106119: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 43. (index mismatch) I0507 06:34:21.646127 16769 consensus_queue.cc:984] T 1119750e874a4ae1b742612a6803ea81 P 40616cbd3b6a40bd860286f0be106119 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48ea1474af5442249334996e0babce18" member_type: NON_VOTER last_known_addr { host: "127.0.57.195" port: 37195 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 43, Last known committed idx: 1, Time since last communication: 0.000s I0507 06:34:21.646374 16769 consensus_queue.cc:689] T 1119750e874a4ae1b742612a6803ea81 P 40616cbd3b6a40bd860286f0be106119 [LEADER]: The logs necessary to catch up peer 48ea1474af5442249334996e0babce18 have been garbage collected. The follower will never be able to catch up (Not found: Failed to read ops 2..42: Segment 1 which contained index 2 has been GCed) I0507 06:34:21.646502 16769 consensus_queue.cc:544] T 1119750e874a4ae1b742612a6803ea81 P 40616cbd3b6a40bd860286f0be106119 [LEADER]: The logs necessary to catch up peer 48ea1474af5442249334996e0babce18 have been garbage collected. The replica will never be able to catch up I0507 06:34:21.647313 16912 raft_consensus.cc:2788] T 1119750e874a4ae1b742612a6803ea81 P 40616cbd3b6a40bd860286f0be106119 [term 1 LEADER]: Committing config change with OpId 1.43: config changed from index 42 to 43, NON_VOTER 48ea1474af5442249334996e0babce18 (127.0.57.195) added. New config: { opid_index: 43 OBSOLETE_local: false peers { permanent_uuid: "58f3e2c789b64bb0a81cc10c779b95e3" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 38173 } } peers { permanent_uuid: "40616cbd3b6a40bd860286f0be106119" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 43553 } } peers { permanent_uuid: "48ea1474af5442249334996e0babce18" member_type: NON_VOTER last_known_addr { host: "127.0.57.195" port: 37195 } attrs { promote: true } } } I0507 06:34:21.647593 16839 tablet_service.cc:1057] Processing DeleteTablet for tablet 1119750e874a4ae1b742612a6803ea81 with delete_type TABLET_DATA_TOMBSTONED (TS 48ea1474af5442249334996e0babce18 not found in new config with opid_index 42) from {username='slave'} at 127.0.0.1:51492 I0507 06:34:21.650256 16249 catalog_manager.cc:3531] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 1119750e874a4ae1b742612a6803ea81 with cas_config_opid_index 42: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1) I0507 06:34:21.651142 16263 catalog_manager.cc:3988] T 1119750e874a4ae1b742612a6803ea81 P 40616cbd3b6a40bd860286f0be106119 reported cstate change: config changed from index 42 to 43, NON_VOTER 48ea1474af5442249334996e0babce18 (127.0.57.195) added. New cstate: current_term: 1 leader_uuid: "40616cbd3b6a40bd860286f0be106119" committed_config { opid_index: 43 OBSOLETE_local: false peers { permanent_uuid: "58f3e2c789b64bb0a81cc10c779b95e3" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 38173 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "40616cbd3b6a40bd860286f0be106119" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 43553 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "48ea1474af5442249334996e0babce18" member_type: NON_VOTER last_known_addr { host: "127.0.57.195" port: 37195 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } } I0507 06:34:21.651561 16509 raft_consensus.cc:2788] T 1119750e874a4ae1b742612a6803ea81 P 58f3e2c789b64bb0a81cc10c779b95e3 [term 1 FOLLOWER]: Committing config change with OpId 1.43: config changed from index 42 to 43, NON_VOTER 48ea1474af5442249334996e0babce18 (127.0.57.195) added. New config: { opid_index: 43 OBSOLETE_local: false peers { permanent_uuid: "58f3e2c789b64bb0a81cc10c779b95e3" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 38173 } } peers { permanent_uuid: "40616cbd3b6a40bd860286f0be106119" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 43553 } } peers { permanent_uuid: "48ea1474af5442249334996e0babce18" member_type: NON_VOTER last_known_addr { host: "127.0.57.195" port: 37195 } attrs { promote: true } } } I0507 06:34:21.653108 16919 tablet_replica.cc:264] T 1119750e874a4ae1b742612a6803ea81 P 48ea1474af5442249334996e0babce18: stopping tablet replica I0507 06:34:21.653532 16919 raft_consensus.cc:2143] T 1119750e874a4ae1b742612a6803ea81 P 48ea1474af5442249334996e0babce18 [term 1 FOLLOWER]: Raft consensus shutting down. I0507 06:34:21.653929 16919 raft_consensus.cc:2170] T 1119750e874a4ae1b742612a6803ea81 P 48ea1474af5442249334996e0babce18 [term 1 FOLLOWER]: Raft consensus is shut down! I0507 06:34:21.655879 16263 catalog_manager.cc:3739] Sending ChangeConfig:REMOVE_PEER on tablet 1119750e874a4ae1b742612a6803ea81 (attempt 1) I0507 06:34:21.655977 16919 ts_tablet_manager.cc:1377] T 1119750e874a4ae1b742612a6803ea81 P 48ea1474af5442249334996e0babce18: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED I0507 06:34:21.657258 16385 consensus_queue.cc:206] T 1119750e874a4ae1b742612a6803ea81 P 40616cbd3b6a40bd860286f0be106119 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 43, Committed index: 43, Last appended: 1.43, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 44 OBSOLETE_local: false peers { permanent_uuid: "58f3e2c789b64bb0a81cc10c779b95e3" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 38173 } } peers { permanent_uuid: "40616cbd3b6a40bd860286f0be106119" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 43553 } } I0507 06:34:21.658680 16509 raft_consensus.cc:1181] T 1119750e874a4ae1b742612a6803ea81 P 58f3e2c789b64bb0a81cc10c779b95e3 [term 1 FOLLOWER]: Refusing update from remote peer 40616cbd3b6a40bd860286f0be106119: Log matching property violated. Preceding OpId in replica: term: 1 index: 43. Preceding OpId from leader: term: 1 index: 44. (index mismatch) I0507 06:34:21.659130 16769 consensus_queue.cc:984] T 1119750e874a4ae1b742612a6803ea81 P 40616cbd3b6a40bd860286f0be106119 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58f3e2c789b64bb0a81cc10c779b95e3" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 38173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 44, Last known committed idx: 43, Time since last communication: 0.000s I0507 06:34:21.661012 16912 raft_consensus.cc:2788] T 1119750e874a4ae1b742612a6803ea81 P 40616cbd3b6a40bd860286f0be106119 [term 1 LEADER]: Committing config change with OpId 1.44: config changed from index 43 to 44, NON_VOTER 48ea1474af5442249334996e0babce18 (127.0.57.195) evicted. New config: { opid_index: 44 OBSOLETE_local: false peers { permanent_uuid: "58f3e2c789b64bb0a81cc10c779b95e3" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 38173 } } peers { permanent_uuid: "40616cbd3b6a40bd860286f0be106119" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 43553 } } } I0507 06:34:21.661604 16509 raft_consensus.cc:2788] T 1119750e874a4ae1b742612a6803ea81 P 58f3e2c789b64bb0a81cc10c779b95e3 [term 1 FOLLOWER]: Committing config change with OpId 1.44: config changed from index 43 to 44, NON_VOTER 48ea1474af5442249334996e0babce18 (127.0.57.195) evicted. New config: { opid_index: 44 OBSOLETE_local: false peers { permanent_uuid: "58f3e2c789b64bb0a81cc10c779b95e3" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 38173 } } peers { permanent_uuid: "40616cbd3b6a40bd860286f0be106119" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 43553 } } } I0507 06:34:21.663252 16919 ts_tablet_manager.cc:1390] T 1119750e874a4ae1b742612a6803ea81 P 48ea1474af5442249334996e0babce18: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.1 I0507 06:34:21.663419 16919 log.cc:990] T 1119750e874a4ae1b742612a6803ea81 P 48ea1474af5442249334996e0babce18: Deleting WAL directory at /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_5.1557210500371863-231/raft_consensus-itest-cluster/ts-2/wal/wals/1119750e874a4ae1b742612a6803ea81 I0507 06:34:21.663569 16249 catalog_manager.cc:3531] ChangeConfig:REMOVE_PEER RPC for tablet 1119750e874a4ae1b742612a6803ea81 with cas_config_opid_index 43: ChangeConfig:REMOVE_PEER succeeded (attempt 1) I0507 06:34:21.664480 16249 catalog_manager.cc:3368] TS 48ea1474af5442249334996e0babce18 (127.0.57.195:37195): tablet 1119750e874a4ae1b742612a6803ea81 (table TestTable [id=4efac679a2544c17946c9f4b94d0b144]) successfully deleted I0507 06:34:21.664775 16430 maintenance_manager.cc:306] P 40616cbd3b6a40bd860286f0be106119: Scheduling LogGCOp(1119750e874a4ae1b742612a6803ea81): free 1840778 bytes of WAL I0507 06:34:21.665098 16324 log_reader.cc:377] T 1119750e874a4ae1b742612a6803ea81: removed 2 log segments from log reader I0507 06:34:21.664849 16263 catalog_manager.cc:3988] T 1119750e874a4ae1b742612a6803ea81 P 40616cbd3b6a40bd860286f0be106119 reported cstate change: config changed from index 43 to 44, NON_VOTER 48ea1474af5442249334996e0babce18 (127.0.57.195) evicted. New cstate: current_term: 1 leader_uuid: "40616cbd3b6a40bd860286f0be106119" committed_config { opid_index: 44 OBSOLETE_local: false peers { permanent_uuid: "58f3e2c789b64bb0a81cc10c779b95e3" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 38173 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "40616cbd3b6a40bd860286f0be106119" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 43553 } health_report { overall_health: HEALTHY } } } I0507 06:34:21.665172 16324 log.cc:863] T 1119750e874a4ae1b742612a6803ea81 P 40616cbd3b6a40bd860286f0be106119: Deleting log segment in path: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_5.1557210500371863-231/raft_consensus-itest-cluster/ts-0/wal/wals/1119750e874a4ae1b742612a6803ea81/wal-000000004 (ops 23-29) I0507 06:34:21.665239 16324 log.cc:863] T 1119750e874a4ae1b742612a6803ea81 P 40616cbd3b6a40bd860286f0be106119: Deleting log segment in path: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_5.1557210500371863-231/raft_consensus-itest-cluster/ts-0/wal/wals/1119750e874a4ae1b742612a6803ea81/wal-000000005 (ops 30-36) I0507 06:34:21.665801 16324 maintenance_manager.cc:501] P 40616cbd3b6a40bd860286f0be106119: LogGCOp(1119750e874a4ae1b742612a6803ea81) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {} I0507 06:34:21.667850 16263 catalog_manager.cc:3390] Sending DeleteTablet(TABLET_DATA_TOMBSTONED) for tablet 1119750e874a4ae1b742612a6803ea81 on 48ea1474af5442249334996e0babce18 (127.0.57.195:37195) (TS 48ea1474af5442249334996e0babce18 not found in new config with opid_index 44) I0507 06:34:21.668227 16263 catalog_manager.cc:3616] Sending ChangeConfig:ADD_PEER:NON_VOTER on tablet 1119750e874a4ae1b742612a6803ea81 (attempt 1) I0507 06:34:21.668473 16839 tablet_service.cc:1057] Processing DeleteTablet for tablet 1119750e874a4ae1b742612a6803ea81 with delete_type TABLET_DATA_TOMBSTONED (TS 48ea1474af5442249334996e0babce18 not found in new config with opid_index 44) from {username='slave'} at 127.0.0.1:51492 I0507 06:34:21.669106 16249 catalog_manager.cc:3368] TS 48ea1474af5442249334996e0babce18 (127.0.57.195:37195): tablet 1119750e874a4ae1b742612a6803ea81 (table TestTable [id=4efac679a2544c17946c9f4b94d0b144]) successfully deleted I0507 06:34:21.669497 16385 consensus_queue.cc:206] T 1119750e874a4ae1b742612a6803ea81 P 40616cbd3b6a40bd860286f0be106119 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 44, Committed index: 44, Last appended: 1.44, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 45 OBSOLETE_local: false peers { permanent_uuid: "58f3e2c789b64bb0a81cc10c779b95e3" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 38173 } } peers { permanent_uuid: "40616cbd3b6a40bd860286f0be106119" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 43553 } } peers { permanent_uuid: "48ea1474af5442249334996e0babce18" member_type: NON_VOTER last_known_addr { host: "127.0.57.195" port: 37195 } attrs { promote: true } } I0507 06:34:21.671082 16509 raft_consensus.cc:1181] T 1119750e874a4ae1b742612a6803ea81 P 58f3e2c789b64bb0a81cc10c779b95e3 [term 1 FOLLOWER]: Refusing update from remote peer 40616cbd3b6a40bd860286f0be106119: Log matching property violated. Preceding OpId in replica: term: 1 index: 44. Preceding OpId from leader: term: 1 index: 45. (index mismatch) W0507 06:34:21.671478 16320 consensus_peers.cc:457] T 1119750e874a4ae1b742612a6803ea81 P 40616cbd3b6a40bd860286f0be106119 -> Peer 48ea1474af5442249334996e0babce18 (127.0.57.195:37195): Couldn't send request to peer 48ea1474af5442249334996e0babce18. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry. I0507 06:34:21.671525 16769 consensus_queue.cc:984] T 1119750e874a4ae1b742612a6803ea81 P 40616cbd3b6a40bd860286f0be106119 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58f3e2c789b64bb0a81cc10c779b95e3" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 38173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 45, Last known committed idx: 44, Time since last communication: 0.000s I0507 06:34:21.673343 16912 raft_consensus.cc:2788] T 1119750e874a4ae1b742612a6803ea81 P 40616cbd3b6a40bd860286f0be106119 [term 1 LEADER]: Committing config change with OpId 1.45: config changed from index 44 to 45, NON_VOTER 48ea1474af5442249334996e0babce18 (127.0.57.195) added. New config: { opid_index: 45 OBSOLETE_local: false peers { permanent_uuid: "58f3e2c789b64bb0a81cc10c779b95e3" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 38173 } } peers { permanent_uuid: "40616cbd3b6a40bd860286f0be106119" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 43553 } } peers { permanent_uuid: "48ea1474af5442249334996e0babce18" member_type: NON_VOTER last_known_addr { host: "127.0.57.195" port: 37195 } attrs { promote: true } } } I0507 06:34:21.673864 16509 raft_consensus.cc:2788] T 1119750e874a4ae1b742612a6803ea81 P 58f3e2c789b64bb0a81cc10c779b95e3 [term 1 FOLLOWER]: Committing config change with OpId 1.45: config changed from index 44 to 45, NON_VOTER 48ea1474af5442249334996e0babce18 (127.0.57.195) added. New config: { opid_index: 45 OBSOLETE_local: false peers { permanent_uuid: "58f3e2c789b64bb0a81cc10c779b95e3" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 38173 } } peers { permanent_uuid: "40616cbd3b6a40bd860286f0be106119" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 43553 } } peers { permanent_uuid: "48ea1474af5442249334996e0babce18" member_type: NON_VOTER last_known_addr { host: "127.0.57.195" port: 37195 } attrs { promote: true } } } I0507 06:34:21.675885 16249 catalog_manager.cc:3531] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 1119750e874a4ae1b742612a6803ea81 with cas_config_opid_index 44: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1) I0507 06:34:21.676837 16263 catalog_manager.cc:3988] T 1119750e874a4ae1b742612a6803ea81 P 40616cbd3b6a40bd860286f0be106119 reported cstate change: config changed from index 44 to 45, NON_VOTER 48ea1474af5442249334996e0babce18 (127.0.57.195) added. New cstate: current_term: 1 leader_uuid: "40616cbd3b6a40bd860286f0be106119" committed_config { opid_index: 45 OBSOLETE_local: false peers { permanent_uuid: "58f3e2c789b64bb0a81cc10c779b95e3" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 38173 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "40616cbd3b6a40bd860286f0be106119" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 43553 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "48ea1474af5442249334996e0babce18" member_type: NON_VOTER last_known_addr { host: "127.0.57.195" port: 37195 } attrs { promote: true } health_report { overall_health: UNKNOWN } } } I0507 06:34:21.823798 16249 catalog_manager.cc:3390] Sending DeleteTablet(TABLET_DATA_TOMBSTONED) for tablet 1119750e874a4ae1b742612a6803ea81 on 48ea1474af5442249334996e0babce18 (127.0.57.195:37195) (TS 48ea1474af5442249334996e0babce18 not found in new config with opid_index 40) I0507 06:34:21.824848 16839 tablet_service.cc:1057] Processing DeleteTablet for tablet 1119750e874a4ae1b742612a6803ea81 with delete_type TABLET_DATA_TOMBSTONED (TS 48ea1474af5442249334996e0babce18 not found in new config with opid_index 40) from {username='slave'} at 127.0.0.1:51492 I0507 06:34:21.825577 16249 catalog_manager.cc:3368] TS 48ea1474af5442249334996e0babce18 (127.0.57.195:37195): tablet 1119750e874a4ae1b742612a6803ea81 (table TestTable [id=4efac679a2544c17946c9f4b94d0b144]) successfully deleted I0507 06:34:22.226477 16921 ts_tablet_manager.cc:676] T 1119750e874a4ae1b742612a6803ea81 P 48ea1474af5442249334996e0babce18: Initiating tablet copy from peer 40616cbd3b6a40bd860286f0be106119 (127.0.57.193:43553) I0507 06:34:22.226914 16921 tablet_copy_client.cc:240] T 1119750e874a4ae1b742612a6803ea81 P 48ea1474af5442249334996e0babce18: tablet copy: Beginning tablet copy session from remote peer at address 127.0.57.193:43553 I0507 06:34:22.233364 16406 tablet_copy_service.cc:135] P 40616cbd3b6a40bd860286f0be106119: Received BeginTabletCopySession request for tablet 1119750e874a4ae1b742612a6803ea81 from peer 48ea1474af5442249334996e0babce18 ({username='slave'} at 127.0.57.195:36017) I0507 06:34:22.233557 16406 tablet_copy_service.cc:156] P 40616cbd3b6a40bd860286f0be106119: Beginning new tablet copy session on tablet 1119750e874a4ae1b742612a6803ea81 from peer 48ea1474af5442249334996e0babce18 at {username='slave'} at 127.0.57.195:36017: session id = 48ea1474af5442249334996e0babce18-1119750e874a4ae1b742612a6803ea81 I0507 06:34:22.236236 16406 tablet_copy_source_session.cc:202] T 1119750e874a4ae1b742612a6803ea81 P 40616cbd3b6a40bd860286f0be106119: Tablet Copy: opened 5 blocks and 1 log segments I0507 06:34:22.238143 16921 ts_tablet_manager.cc:1377] T 1119750e874a4ae1b742612a6803ea81 P 48ea1474af5442249334996e0babce18: Deleting tablet data with delete state TABLET_DATA_COPYING I0507 06:34:22.242023 16921 ts_tablet_manager.cc:1390] T 1119750e874a4ae1b742612a6803ea81 P 48ea1474af5442249334996e0babce18: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.1 I0507 06:34:22.242377 16921 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet 1119750e874a4ae1b742612a6803ea81. 1 dirs total, 0 dirs full, 0 dirs failed I0507 06:34:22.244585 16921 tablet_copy_client.cc:575] T 1119750e874a4ae1b742612a6803ea81 P 48ea1474af5442249334996e0babce18: tablet copy: Starting download of 5 data blocks... I0507 06:34:22.253959 16921 tablet_copy_client.cc:538] T 1119750e874a4ae1b742612a6803ea81 P 48ea1474af5442249334996e0babce18: tablet copy: Starting download of 1 WAL segments... I0507 06:34:22.258127 16921 tablet_copy_client.cc:414] T 1119750e874a4ae1b742612a6803ea81 P 48ea1474af5442249334996e0babce18: tablet copy: Tablet Copy complete. Replacing tablet superblock. I0507 06:34:22.261209 16921 tablet_bootstrap.cc:439] T 1119750e874a4ae1b742612a6803ea81 P 48ea1474af5442249334996e0babce18: Bootstrap starting. I0507 06:34:22.282048 16921 tablet_bootstrap.cc:439] T 1119750e874a4ae1b742612a6803ea81 P 48ea1474af5442249334996e0babce18: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=1} inserts{seen=2 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=3. Pending: 0 replicates I0507 06:34:22.282496 16921 tablet_bootstrap.cc:439] T 1119750e874a4ae1b742612a6803ea81 P 48ea1474af5442249334996e0babce18: Bootstrap complete. I0507 06:34:22.282856 16921 ts_tablet_manager.cc:1053] T 1119750e874a4ae1b742612a6803ea81 P 48ea1474af5442249334996e0babce18: Time spent bootstrapping tablet: real 0.022s user 0.019s sys 0.000s I0507 06:34:22.283725 16921 raft_consensus.cc:340] T 1119750e874a4ae1b742612a6803ea81 P 48ea1474af5442249334996e0babce18 [term 1 LEARNER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: 45 OBSOLETE_local: false peers { permanent_uuid: "58f3e2c789b64bb0a81cc10c779b95e3" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 38173 } } peers { permanent_uuid: "40616cbd3b6a40bd860286f0be106119" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 43553 } } peers { permanent_uuid: "48ea1474af5442249334996e0babce18" member_type: NON_VOTER last_known_addr { host: "127.0.57.195" port: 37195 } attrs { promote: true } } I0507 06:34:22.283928 16921 raft_consensus.cc:705] T 1119750e874a4ae1b742612a6803ea81 P 48ea1474af5442249334996e0babce18 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 48ea1474af5442249334996e0babce18, State: Initialized, Role: LEARNER I0507 06:34:22.284075 16921 consensus_queue.cc:229] T 1119750e874a4ae1b742612a6803ea81 P 48ea1474af5442249334996e0babce18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 45, Last appended: 1.45, Last appended by leader: 45, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 45 OBSOLETE_local: false peers { permanent_uuid: "58f3e2c789b64bb0a81cc10c779b95e3" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 38173 } } peers { permanent_uuid: "40616cbd3b6a40bd860286f0be106119" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 43553 } } peers { permanent_uuid: "48ea1474af5442249334996e0babce18" member_type: NON_VOTER last_known_addr { host: "127.0.57.195" port: 37195 } attrs { promote: true } } I0507 06:34:22.284879 16903 heartbeater.cc:467] Master 127.0.57.254:38831 was elected leader, sending a full tablet report... I0507 06:34:22.284999 16921 ts_tablet_manager.cc:1081] T 1119750e874a4ae1b742612a6803ea81 P 48ea1474af5442249334996e0babce18: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s I0507 06:34:22.286025 16406 tablet_copy_service.cc:325] P 40616cbd3b6a40bd860286f0be106119: Request end of tablet copy session 48ea1474af5442249334996e0babce18-1119750e874a4ae1b742612a6803ea81 received from {username='slave'} at 127.0.57.195:36017 I0507 06:34:22.286175 16406 tablet_copy_service.cc:413] P 40616cbd3b6a40bd860286f0be106119: ending tablet copy session 48ea1474af5442249334996e0babce18-1119750e874a4ae1b742612a6803ea81 on tablet 1119750e874a4ae1b742612a6803ea81 with peer 48ea1474af5442249334996e0babce18 I0507 06:34:22.377077 16904 maintenance_manager.cc:306] P 48ea1474af5442249334996e0babce18: Scheduling UndoDeltaBlockGCOp(1119750e874a4ae1b742612a6803ea81): 536 bytes on disk I0507 06:34:22.378914 16795 maintenance_manager.cc:501] P 48ea1474af5442249334996e0babce18: UndoDeltaBlockGCOp(1119750e874a4ae1b742612a6803ea81) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":107,"lbm_reads_lt_1ms":4} I0507 06:34:22.708820 16859 raft_consensus.cc:1126] T 1119750e874a4ae1b742612a6803ea81 P 48ea1474af5442249334996e0babce18 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.44->[1.45-1.45] Dedup: 1.45->[] I0507 06:34:22.769888 16553 maintenance_manager.cc:306] P 58f3e2c789b64bb0a81cc10c779b95e3: Scheduling LogGCOp(1119750e874a4ae1b742612a6803ea81): free 1709271 bytes of WAL I0507 06:34:22.770347 16447 log_reader.cc:377] T 1119750e874a4ae1b742612a6803ea81: removed 2 log segments from log reader I0507 06:34:22.770418 16447 log.cc:863] T 1119750e874a4ae1b742612a6803ea81 P 58f3e2c789b64bb0a81cc10c779b95e3: Deleting log segment in path: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_5.1557210500371863-231/raft_consensus-itest-cluster/ts-1/wal/wals/1119750e874a4ae1b742612a6803ea81/wal-000000004 (ops 21-27) I0507 06:34:22.770534 16447 log.cc:863] T 1119750e874a4ae1b742612a6803ea81 P 58f3e2c789b64bb0a81cc10c779b95e3: Deleting log segment in path: /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_5.1557210500371863-231/raft_consensus-itest-cluster/ts-1/wal/wals/1119750e874a4ae1b742612a6803ea81/wal-000000005 (ops 28-33) I0507 06:34:22.771025 16447 maintenance_manager.cc:501] P 58f3e2c789b64bb0a81cc10c779b95e3: LogGCOp(1119750e874a4ae1b742612a6803ea81) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {} I0507 06:34:23.183229 16926 raft_consensus.cc:974] T 1119750e874a4ae1b742612a6803ea81 P 40616cbd3b6a40bd860286f0be106119: attempting to promote NON_VOTER 48ea1474af5442249334996e0babce18 to VOTER I0507 06:34:23.183903 16926 consensus_queue.cc:206] T 1119750e874a4ae1b742612a6803ea81 P 40616cbd3b6a40bd860286f0be106119 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 45, Committed index: 45, Last appended: 1.45, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 46 OBSOLETE_local: false peers { permanent_uuid: "58f3e2c789b64bb0a81cc10c779b95e3" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 38173 } } peers { permanent_uuid: "40616cbd3b6a40bd860286f0be106119" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 43553 } } peers { permanent_uuid: "48ea1474af5442249334996e0babce18" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 37195 } attrs { promote: false } } I0507 06:34:23.186097 16859 raft_consensus.cc:1181] T 1119750e874a4ae1b742612a6803ea81 P 48ea1474af5442249334996e0babce18 [term 1 LEARNER]: Refusing update from remote peer 40616cbd3b6a40bd860286f0be106119: Log matching property violated. Preceding OpId in replica: term: 1 index: 45. Preceding OpId from leader: term: 1 index: 46. (index mismatch) I0507 06:34:23.186286 16509 raft_consensus.cc:1181] T 1119750e874a4ae1b742612a6803ea81 P 58f3e2c789b64bb0a81cc10c779b95e3 [term 1 FOLLOWER]: Refusing update from remote peer 40616cbd3b6a40bd860286f0be106119: Log matching property violated. Preceding OpId in replica: term: 1 index: 45. Preceding OpId from leader: term: 1 index: 46. (index mismatch) I0507 06:34:23.186640 16920 consensus_queue.cc:984] T 1119750e874a4ae1b742612a6803ea81 P 40616cbd3b6a40bd860286f0be106119 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48ea1474af5442249334996e0babce18" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 37195 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 46, Last known committed idx: 45, Time since last communication: 0.000s I0507 06:34:23.186936 16927 consensus_queue.cc:984] T 1119750e874a4ae1b742612a6803ea81 P 40616cbd3b6a40bd860286f0be106119 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58f3e2c789b64bb0a81cc10c779b95e3" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 38173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 46, Last known committed idx: 45, Time since last communication: 0.000s I0507 06:34:23.190361 16927 raft_consensus.cc:2788] T 1119750e874a4ae1b742612a6803ea81 P 40616cbd3b6a40bd860286f0be106119 [term 1 LEADER]: Committing config change with OpId 1.46: config changed from index 45 to 46, 48ea1474af5442249334996e0babce18 (127.0.57.195) changed from NON_VOTER to VOTER. New config: { opid_index: 46 OBSOLETE_local: false peers { permanent_uuid: "58f3e2c789b64bb0a81cc10c779b95e3" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 38173 } } peers { permanent_uuid: "40616cbd3b6a40bd860286f0be106119" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 43553 } } peers { permanent_uuid: "48ea1474af5442249334996e0babce18" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 37195 } attrs { promote: false } } } I0507 06:34:23.190956 16509 raft_consensus.cc:2788] T 1119750e874a4ae1b742612a6803ea81 P 58f3e2c789b64bb0a81cc10c779b95e3 [term 1 FOLLOWER]: Committing config change with OpId 1.46: config changed from index 45 to 46, 48ea1474af5442249334996e0babce18 (127.0.57.195) changed from NON_VOTER to VOTER. New config: { opid_index: 46 OBSOLETE_local: false peers { permanent_uuid: "58f3e2c789b64bb0a81cc10c779b95e3" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 38173 } } peers { permanent_uuid: "40616cbd3b6a40bd860286f0be106119" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 43553 } } peers { permanent_uuid: "48ea1474af5442249334996e0babce18" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 37195 } attrs { promote: false } } } I0507 06:34:23.191380 16859 raft_consensus.cc:2788] T 1119750e874a4ae1b742612a6803ea81 P 48ea1474af5442249334996e0babce18 [term 1 FOLLOWER]: Committing config change with OpId 1.46: config changed from index 45 to 46, 48ea1474af5442249334996e0babce18 (127.0.57.195) changed from NON_VOTER to VOTER. New config: { opid_index: 46 OBSOLETE_local: false peers { permanent_uuid: "58f3e2c789b64bb0a81cc10c779b95e3" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 38173 } } peers { permanent_uuid: "40616cbd3b6a40bd860286f0be106119" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 43553 } } peers { permanent_uuid: "48ea1474af5442249334996e0babce18" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 37195 } attrs { promote: false } } } I0507 06:34:23.195261 16262 catalog_manager.cc:3988] T 1119750e874a4ae1b742612a6803ea81 P 40616cbd3b6a40bd860286f0be106119 reported cstate change: config changed from index 45 to 46, 48ea1474af5442249334996e0babce18 (127.0.57.195) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "40616cbd3b6a40bd860286f0be106119" committed_config { opid_index: 46 OBSOLETE_local: false peers { permanent_uuid: "58f3e2c789b64bb0a81cc10c779b95e3" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 38173 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "40616cbd3b6a40bd860286f0be106119" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 43553 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "48ea1474af5442249334996e0babce18" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 37195 } attrs { promote: false } health_report { overall_health: HEALTHY } } } Master Summary UUID | Address | Status ----------------------------------+--------------------+--------- 3d5e6cea88e548a1b14d0e4a8455d204 | 127.0.57.254:38831 | HEALTHY Flag | Value | Tags | Master ------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+------------------------- ipki_ca_key_size | 1024 | experimental | all 1 server(s) checked ipki_server_key_size | 1024 | experimental | all 1 server(s) checked never_fsync | true | unsafe,advanced | all 1 server(s) checked raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked rpc_reuseport | true | experimental | all 1 server(s) checked rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked server_dump_info_format | pb | hidden | all 1 server(s) checked server_dump_info_path | /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_5.1557210500371863-231/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked Tablet Server Summary UUID | Address | Status | Location ----------------------------------+--------------------+---------+---------- 40616cbd3b6a40bd860286f0be106119 | 127.0.57.193:43553 | HEALTHY | 48ea1474af5442249334996e0babce18 | 127.0.57.195:37195 | HEALTHY | 58f3e2c789b64bb0a81cc10c779b95e3 | 127.0.57.194:38173 | HEALTHY | Tablet Server Location Summary Location | Count ----------+--------- | 3 Flag | Value | Tags | Tablet Server ------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------------------------------+------------------------- ipki_server_key_size | 1024 | experimental | all 3 server(s) checked local_ip_for_outbound_sockets | 127.0.57.193 | experimental | 127.0.57.193:43553 local_ip_for_outbound_sockets | 127.0.57.194 | experimental | 127.0.57.194:38173 local_ip_for_outbound_sockets | 127.0.57.195 | experimental | 127.0.57.195:37195 log_compression_codec | none | experimental | all 3 server(s) checked log_max_segments_to_retain | 3 | experimental,advanced,runtime | all 3 server(s) checked log_target_replay_size_mb | 1 | experimental | all 3 server(s) checked maintenance_manager_polling_interval_ms | 100 | hidden | all 3 server(s) checked max_cell_size_bytes | 1000000 | unsafe | all 3 server(s) checked never_fsync | true | unsafe,advanced | all 3 server(s) checked raft_prepare_replacement_before_eviction | true | experimental,advanced | all 3 server(s) checked rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked server_dump_info_format | pb | hidden | all 3 server(s) checked server_dump_info_path | /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_5.1557210500371863-231/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.0.57.193:43553 server_dump_info_path | /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_5.1557210500371863-231/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.0.57.194:38173 server_dump_info_path | /tmp/dist-test-taskKhSYji/test-tmp/raft_consensus_nonvoter-itest.0.ReplicaBehindWalGcThresholdITest.ReplicaReplacement_5.1557210500371863-231/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.0.57.195:37195 Version Summary Version | Servers ------------------------+------------------------- 1.10.0-cdh6.x-SNAPSHOT | all 4 server(s) checked Tablet Summary Summary by table Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable -----------+----+---------+---------------+---------+------------+------------------+------------- TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0 Tablet Replica Count Summary Statistic | Replica Count ----------------+--------------- Minimum | 1 First Quartile | 1 Median | 1 Third Quartile | 1 Maximum | 1 Total Count Summary | Total Count ----------------+------------- Masters | 1 Tablet Servers | 3 Tables | 1 Tablets | 1 Replicas | 3 ================== Warnings: ================== Some masters have unsafe, experimental, or hidden flags set Some tablet servers have unsafe, experimental, or hidden flags set OK I0507 06:34:23.308785 231 log_verifier.cc:121] Checking tablet 1119750e874a4ae1b742612a6803ea81 I0507 06:34:23.334173 231 log_verifier.cc:172] Verified matching terms for 13 ops in tablet 1119750e874a4ae1b742612a6803ea81 I0507 06:34:23.335803 231 external_mini_cluster.cc:1172] Attempting to check leaks for /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver pid 16300 I0507 06:34:23.337221 16426 generic_service.cc:171] Checking for leaks (request via RPC) I0507 06:34:23.440881 231 external_mini_cluster.cc:1131] Killing /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver with pid 16300 I0507 06:34:23.459316 231 external_mini_cluster.cc:1172] Attempting to check leaks for /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver pid 16433 I0507 06:34:23.460341 16549 generic_service.cc:171] Checking for leaks (request via RPC) I0507 06:34:23.554723 231 external_mini_cluster.cc:1131] Killing /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver with pid 16433 I0507 06:34:23.570832 231 external_mini_cluster.cc:1172] Attempting to check leaks for /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver pid 16781 I0507 06:34:23.575954 16899 generic_service.cc:171] Checking for leaks (request via RPC) I0507 06:34:23.631496 231 external_mini_cluster.cc:1131] Killing /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-tserver with pid 16781 I0507 06:34:23.641243 231 external_mini_cluster.cc:1172] Attempting to check leaks for /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-master pid 16239 I0507 06:34:23.642283 16295 generic_service.cc:171] Checking for leaks (request via RPC) I0507 06:34:23.732960 231 external_mini_cluster.cc:1131] Killing /tmp/dist-test-taskKhSYji/build/asan/bin/kudu-master with pid 16239 [ OK ] ReplicaBehindWalGcThresholdITest.ReplicaReplacement/5 (22208 ms) [----------] 6 tests from ReplicaBehindWalGcThresholdITest (93764 ms total) [----------] Global test environment tear-down [==========] 23 tests from 3 test cases ran. (363331 ms total) [ PASSED ] 22 tests. [ FAILED ] 1 test, listed below: [ FAILED ] RaftConsensusNonVoterITest.PromoteAndDemote 1 FAILED TEST