[ RUN ] RaftConsensusNonVoterITest.PromoteAndDemote I0504 12:39:09.148201 231 test_util.cc:212] Using random seed: 659707927 I0504 12:39:09.149637 231 ts_itest-base.cc:112] Starting cluster with: I0504 12:39:09.149679 231 ts_itest-base.cc:113] -------------- I0504 12:39:09.149713 231 ts_itest-base.cc:114] 4 tablet servers I0504 12:39:09.149730 231 ts_itest-base.cc:115] 3 replicas per TS I0504 12:39:09.149758 231 ts_itest-base.cc:116] -------------- I0504 12:39:09.150954 231 external_mini_cluster.cc:933] Running /tmp/dist-test-taskHOObI0/build/asan/bin/kudu-master /tmp/dist-test-taskHOObI0/build/asan/bin/kudu-master --fs_wal_dir=/tmp/dist-test-taskHOObI0/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromoteAndDemote.1556973524314425-231/raft_consensus-itest-cluster/master-0/wal --fs_data_dirs=/tmp/dist-test-taskHOObI0/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromoteAndDemote.1556973524314425-231/raft_consensus-itest-cluster/master-0/data --block_manager=log --webserver_interface=localhost --ipki_ca_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.0.57.254:43043 --webserver_interface=127.0.57.254 --webserver_port=0 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskHOObI0/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromoteAndDemote.1556973524314425-231/raft_consensus-itest-cluster/master-0/logs --server_dump_info_path=/tmp/dist-test-taskHOObI0/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromoteAndDemote.1556973524314425-231/raft_consensus-itest-cluster/master-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --catalog_manager_wait_for_new_tablets_to_elect_leader=false --catalog_manager_evict_excess_replicas=false --rpc_reuseport=true with env {} WARNING: Logging before InitGoogleLogging() is written to STDERR W0504 12:39:09.335239 3308 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0504 12:39:09.335492 3308 flags.cc:404] Enabled unsafe flag: --never_fsync=true W0504 12:39:09.337908 3308 flags.cc:404] Enabled experimental flag: --ipki_ca_key_size=1024 W0504 12:39:09.337970 3308 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W0504 12:39:09.338008 3308 flags.cc:404] Enabled experimental flag: --tsk_num_rsa_bits=512 W0504 12:39:09.338048 3308 flags.cc:404] Enabled experimental flag: --rpc_reuseport=true I0504 12:39:09.342469 3308 master_main.cc:73] Master server non-default flags: --fs_data_dirs=/tmp/dist-test-taskHOObI0/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromoteAndDemote.1556973524314425-231/raft_consensus-itest-cluster/master-0/data --fs_wal_dir=/tmp/dist-test-taskHOObI0/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromoteAndDemote.1556973524314425-231/raft_consensus-itest-cluster/master-0/wal --catalog_manager_evict_excess_replicas=false --catalog_manager_wait_for_new_tablets_to_elect_leader=false --ipki_ca_key_size=1024 --ipki_server_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.0.57.254:43043 --rpc_reuseport=true --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskHOObI0/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromoteAndDemote.1556973524314425-231/raft_consensus-itest-cluster/master-0/data/info.pb --webserver_interface=127.0.57.254 --webserver_port=0 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --log_dir=/tmp/dist-test-taskHOObI0/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromoteAndDemote.1556973524314425-231/raft_consensus-itest-cluster/master-0/logs --logbuflevel=-1 --logtostderr=true Master server version: kudu 1.10.0-cdh6.x-SNAPSHOT revision 9f71c857a3824d8541c4608dda2722f71515784f build type FASTDEBUG built by jenkins at 04 May 2019 04:53:16 PST on quasar-btakog-1.vpc.cloudera.com build id 321096 ASAN enabled I0504 12:39:09.348814 3308 system_ntp.cc:202] NTP initialized. Skew: 500ppm Current error: 48016us I0504 12:39:09.350220 3308 fs_manager.cc:263] Metadata directory not provided I0504 12:39:09.350322 3308 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0504 12:39:09.350576 3308 server_base.cc:433] Could not load existing FS layout: Not found: could not find a healthy instance file I0504 12:39:09.350658 3308 server_base.cc:434] Attempting to create new FS layout instead I0504 12:39:09.353036 3308 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskHOObI0/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromoteAndDemote.1556973524314425-231/raft_consensus-itest-cluster/master-0/data/instance: uuid: "3744a79ca8264f1587a6daa834a73948" format_stamp: "Formatted at 2019-05-04 12:39:09 on dist-test-slave-dist-test-slave-jj8h" I0504 12:39:09.353581 3308 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskHOObI0/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromoteAndDemote.1556973524314425-231/raft_consensus-itest-cluster/master-0/wal/instance: uuid: "3744a79ca8264f1587a6daa834a73948" format_stamp: "Formatted at 2019-05-04 12:39:09 on dist-test-slave-dist-test-slave-jj8h" I0504 12:39:09.359877 3308 fs_manager.cc:503] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s I0504 12:39:09.360951 3308 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0504 12:39:09.361510 3308 file_cache.cc:466] Constructed file cache lbm with capacity 419430 I0504 12:39:09.364734 3308 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s I0504 12:39:09.364804 3308 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskHOObI0/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromoteAndDemote.1556973524314425-231/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskHOObI0/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromoteAndDemote.1556973524314425-231/raft_consensus-itest-cluster/master-0/wal uuid: "3744a79ca8264f1587a6daa834a73948" format_stamp: "Formatted at 2019-05-04 12:39:09 on dist-test-slave-dist-test-slave-jj8h" I0504 12:39:09.364965 3308 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskHOObI0/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromoteAndDemote.1556973524314425-231/raft_consensus-itest-cluster/master-0/wal metadata directory: /tmp/dist-test-taskHOObI0/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromoteAndDemote.1556973524314425-231/raft_consensus-itest-cluster/master-0/wal 1 data directories: /tmp/dist-test-taskHOObI0/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromoteAndDemote.1556973524314425-231/raft_consensus-itest-cluster/master-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0504 12:39:09.400574 3308 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0504 12:39:09.415402 3308 rpc_server.cc:205] RPC server started. Bound to: 127.0.57.254:43043 I0504 12:39:09.416704 3308 webserver.cc:308] Webserver started at http://127.0.57.254:38145/ using document root and password file I0504 12:39:09.417546 3308 server_base.cc:608] Dumped server information to /tmp/dist-test-taskHOObI0/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromoteAndDemote.1556973524314425-231/raft_consensus-itest-cluster/master-0/data/info.pb I0504 12:39:09.421136 3367 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0504 12:39:09.424930 231 external_mini_cluster.cc:995] Started /tmp/dist-test-taskHOObI0/build/asan/bin/kudu-master as pid 3308 I0504 12:39:09.425663 231 external_mini_cluster.cc:933] Running /tmp/dist-test-taskHOObI0/build/asan/bin/kudu-tserver /tmp/dist-test-taskHOObI0/build/asan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskHOObI0/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromoteAndDemote.1556973524314425-231/raft_consensus-itest-cluster/ts-0/wal --fs_data_dirs=/tmp/dist-test-taskHOObI0/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromoteAndDemote.1556973524314425-231/raft_consensus-itest-cluster/ts-0/data --block_manager=log --rpc_bind_addresses=127.0.57.193:0 --local_ip_for_outbound_sockets=127.0.57.193 --webserver_interface=127.0.57.193 --webserver_port=0 --tserver_master_addrs=127.0.57.254:43043 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskHOObI0/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromoteAndDemote.1556973524314425-231/raft_consensus-itest-cluster/ts-0/logs --server_dump_info_path=/tmp/dist-test-taskHOObI0/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromoteAndDemote.1556973524314425-231/raft_consensus-itest-cluster/ts-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_heartbeat_interval_ms=50 --evict_failed_followers=false --enable_leader_failure_detection=false with env {} I0504 12:39:09.432910 3367 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 3744a79ca8264f1587a6daa834a73948: Bootstrap starting. I0504 12:39:09.435971 3367 tablet_bootstrap.cc:590] T 00000000000000000000000000000000 P 3744a79ca8264f1587a6daa834a73948: No blocks or log segments found. Creating new log. I0504 12:39:09.436909 3367 log.cc:525] T 00000000000000000000000000000000 P 3744a79ca8264f1587a6daa834a73948: Log is configured to *not* fsync() on all Append() calls I0504 12:39:09.439707 3367 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 3744a79ca8264f1587a6daa834a73948: No bootstrap required, opened a new log I0504 12:39:09.446511 3367 raft_consensus.cc:340] T 00000000000000000000000000000000 P 3744a79ca8264f1587a6daa834a73948 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3744a79ca8264f1587a6daa834a73948" member_type: VOTER } I0504 12:39:09.446749 3367 raft_consensus.cc:366] T 00000000000000000000000000000000 P 3744a79ca8264f1587a6daa834a73948 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0504 12:39:09.446821 3367 raft_consensus.cc:705] T 00000000000000000000000000000000 P 3744a79ca8264f1587a6daa834a73948 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3744a79ca8264f1587a6daa834a73948, State: Initialized, Role: FOLLOWER I0504 12:39:09.447379 3367 consensus_queue.cc:229] T 00000000000000000000000000000000 P 3744a79ca8264f1587a6daa834a73948 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3744a79ca8264f1587a6daa834a73948" member_type: VOTER } I0504 12:39:09.447552 3367 raft_consensus.cc:380] T 00000000000000000000000000000000 P 3744a79ca8264f1587a6daa834a73948 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0504 12:39:09.447643 3367 raft_consensus.cc:472] T 00000000000000000000000000000000 P 3744a79ca8264f1587a6daa834a73948 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0504 12:39:09.447754 3367 raft_consensus.cc:2886] T 00000000000000000000000000000000 P 3744a79ca8264f1587a6daa834a73948 [term 0 FOLLOWER]: Advancing to term 1 I0504 12:39:09.450536 3367 raft_consensus.cc:494] T 00000000000000000000000000000000 P 3744a79ca8264f1587a6daa834a73948 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3744a79ca8264f1587a6daa834a73948" member_type: VOTER } I0504 12:39:09.451050 3367 leader_election.cc:310] T 00000000000000000000000000000000 P 3744a79ca8264f1587a6daa834a73948 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3744a79ca8264f1587a6daa834a73948; no voters: I0504 12:39:09.454557 3367 leader_election.cc:296] T 00000000000000000000000000000000 P 3744a79ca8264f1587a6daa834a73948 [CANDIDATE]: Term 1 election: Requested vote from peers I0504 12:39:09.454717 3370 raft_consensus.cc:2640] T 00000000000000000000000000000000 P 3744a79ca8264f1587a6daa834a73948 [term 1 FOLLOWER]: Leader election won for term 1 I0504 12:39:09.458854 3370 raft_consensus.cc:667] T 00000000000000000000000000000000 P 3744a79ca8264f1587a6daa834a73948 [term 1 LEADER]: Becoming Leader. State: Replica: 3744a79ca8264f1587a6daa834a73948, State: Running, Role: LEADER I0504 12:39:09.459496 3370 consensus_queue.cc:206] T 00000000000000000000000000000000 P 3744a79ca8264f1587a6daa834a73948 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3744a79ca8264f1587a6daa834a73948" member_type: VOTER } I0504 12:39:09.463178 3371 sys_catalog.cc:337] T 00000000000000000000000000000000 P 3744a79ca8264f1587a6daa834a73948 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3744a79ca8264f1587a6daa834a73948" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3744a79ca8264f1587a6daa834a73948" member_type: VOTER } } I0504 12:39:09.463265 3372 sys_catalog.cc:337] T 00000000000000000000000000000000 P 3744a79ca8264f1587a6daa834a73948 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3744a79ca8264f1587a6daa834a73948. Latest consensus state: current_term: 1 leader_uuid: "3744a79ca8264f1587a6daa834a73948" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3744a79ca8264f1587a6daa834a73948" member_type: VOTER } } I0504 12:39:09.463495 3367 sys_catalog.cc:337] T 00000000000000000000000000000000 P 3744a79ca8264f1587a6daa834a73948 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 1 leader_uuid: "3744a79ca8264f1587a6daa834a73948" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3744a79ca8264f1587a6daa834a73948" member_type: VOTER } } I0504 12:39:09.464004 3367 sys_catalog.cc:340] T 00000000000000000000000000000000 P 3744a79ca8264f1587a6daa834a73948 [sys.catalog]: This master's current role is: LEADER I0504 12:39:09.463905 3372 sys_catalog.cc:340] T 00000000000000000000000000000000 P 3744a79ca8264f1587a6daa834a73948 [sys.catalog]: This master's current role is: LEADER I0504 12:39:09.463727 3371 sys_catalog.cc:340] T 00000000000000000000000000000000 P 3744a79ca8264f1587a6daa834a73948 [sys.catalog]: This master's current role is: LEADER I0504 12:39:09.467082 3367 sys_catalog.cc:424] T 00000000000000000000000000000000 P 3744a79ca8264f1587a6daa834a73948 [sys.catalog]: configured and running, proceeding with master startup. I0504 12:39:09.467471 3374 catalog_manager.cc:1055] Loading table and tablet metadata into memory... I0504 12:39:09.473429 3374 catalog_manager.cc:1066] Initializing Kudu internal certificate authority... I0504 12:39:09.568929 3374 catalog_manager.cc:950] Generated new certificate authority record I0504 12:39:09.571027 3374 catalog_manager.cc:1075] Loading token signing keys... I0504 12:39:09.579236 3374 catalog_manager.cc:4291] T 00000000000000000000000000000000 P 3744a79ca8264f1587a6daa834a73948: Generated new TSK 0 WARNING: Logging before InitGoogleLogging() is written to STDERR W0504 12:39:09.590481 3368 flags.cc:404] Enabled unsafe flag: --enable_leader_failure_detection=false W0504 12:39:09.590736 3368 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0504 12:39:09.590816 3368 flags.cc:404] Enabled unsafe flag: --never_fsync=true W0504 12:39:09.593045 3368 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W0504 12:39:09.593158 3368 flags.cc:404] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.57.193 I0504 12:39:09.596884 3368 tablet_server_main.cc:78] Tablet server non-default flags: --enable_leader_failure_detection=false --evict_failed_followers=false --raft_heartbeat_interval_ms=50 --fs_data_dirs=/tmp/dist-test-taskHOObI0/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromoteAndDemote.1556973524314425-231/raft_consensus-itest-cluster/ts-0/data --fs_wal_dir=/tmp/dist-test-taskHOObI0/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromoteAndDemote.1556973524314425-231/raft_consensus-itest-cluster/ts-0/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.0.57.193:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskHOObI0/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromoteAndDemote.1556973524314425-231/raft_consensus-itest-cluster/ts-0/data/info.pb --webserver_interface=127.0.57.193 --webserver_port=0 --tserver_master_addrs=127.0.57.254:43043 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.0.57.193 --log_dir=/tmp/dist-test-taskHOObI0/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromoteAndDemote.1556973524314425-231/raft_consensus-itest-cluster/ts-0/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.10.0-cdh6.x-SNAPSHOT revision 9f71c857a3824d8541c4608dda2722f71515784f build type FASTDEBUG built by jenkins at 04 May 2019 04:53:16 PST on quasar-btakog-1.vpc.cloudera.com build id 321096 ASAN enabled I0504 12:39:09.604466 3368 system_ntp.cc:202] NTP initialized. Skew: 500ppm Current error: 48016us I0504 12:39:09.605221 3368 fs_manager.cc:263] Metadata directory not provided I0504 12:39:09.605288 3368 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0504 12:39:09.605554 3368 server_base.cc:433] Could not load existing FS layout: Not found: could not find a healthy instance file I0504 12:39:09.605617 3368 server_base.cc:434] Attempting to create new FS layout instead I0504 12:39:09.608047 3368 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskHOObI0/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromoteAndDemote.1556973524314425-231/raft_consensus-itest-cluster/ts-0/data/instance: uuid: "1db2335bd82b489ea0f4303451133b62" format_stamp: "Formatted at 2019-05-04 12:39:09 on dist-test-slave-dist-test-slave-jj8h" I0504 12:39:09.608553 3368 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskHOObI0/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromoteAndDemote.1556973524314425-231/raft_consensus-itest-cluster/ts-0/wal/instance: uuid: "1db2335bd82b489ea0f4303451133b62" format_stamp: "Formatted at 2019-05-04 12:39:09 on dist-test-slave-dist-test-slave-jj8h" I0504 12:39:09.625219 3368 fs_manager.cc:503] Time spent creating directory manager: real 0.016s user 0.009s sys 0.008s I0504 12:39:09.626838 3368 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0504 12:39:09.627625 3368 file_cache.cc:466] Constructed file cache lbm with capacity 419430 I0504 12:39:09.631709 3368 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s I0504 12:39:09.631817 3368 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskHOObI0/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromoteAndDemote.1556973524314425-231/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskHOObI0/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromoteAndDemote.1556973524314425-231/raft_consensus-itest-cluster/ts-0/wal uuid: "1db2335bd82b489ea0f4303451133b62" format_stamp: "Formatted at 2019-05-04 12:39:09 on dist-test-slave-dist-test-slave-jj8h" I0504 12:39:09.631996 3368 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskHOObI0/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromoteAndDemote.1556973524314425-231/raft_consensus-itest-cluster/ts-0/wal metadata directory: /tmp/dist-test-taskHOObI0/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromoteAndDemote.1556973524314425-231/raft_consensus-itest-cluster/ts-0/wal 1 data directories: /tmp/dist-test-taskHOObI0/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromoteAndDemote.1556973524314425-231/raft_consensus-itest-cluster/ts-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0504 12:39:09.664691 3368 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0504 12:39:09.679451 3368 ts_tablet_manager.cc:356] Loaded tablet metadata (0 total tablets, 0 live tablets) I0504 12:39:09.679550 3368 ts_tablet_manager.cc:376] Registered 0 tablets I0504 12:39:09.788992 3368 rpc_server.cc:205] RPC server started. Bound to: 127.0.57.193:37541 I0504 12:39:09.790132 3368 webserver.cc:308] Webserver started at http://127.0.57.193:36683/ using document root and password file I0504 12:39:09.790833 3368 server_base.cc:608] Dumped server information to /tmp/dist-test-taskHOObI0/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromoteAndDemote.1556973524314425-231/raft_consensus-itest-cluster/ts-0/data/info.pb I0504 12:39:09.792476 231 external_mini_cluster.cc:995] Started /tmp/dist-test-taskHOObI0/build/asan/bin/kudu-tserver as pid 3368 I0504 12:39:09.793169 231 external_mini_cluster.cc:933] Running /tmp/dist-test-taskHOObI0/build/asan/bin/kudu-tserver /tmp/dist-test-taskHOObI0/build/asan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskHOObI0/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromoteAndDemote.1556973524314425-231/raft_consensus-itest-cluster/ts-1/wal --fs_data_dirs=/tmp/dist-test-taskHOObI0/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromoteAndDemote.1556973524314425-231/raft_consensus-itest-cluster/ts-1/data --block_manager=log --rpc_bind_addresses=127.0.57.194:0 --local_ip_for_outbound_sockets=127.0.57.194 --webserver_interface=127.0.57.194 --webserver_port=0 --tserver_master_addrs=127.0.57.254:43043 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskHOObI0/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromoteAndDemote.1556973524314425-231/raft_consensus-itest-cluster/ts-1/logs --server_dump_info_path=/tmp/dist-test-taskHOObI0/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromoteAndDemote.1556973524314425-231/raft_consensus-itest-cluster/ts-1/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_heartbeat_interval_ms=50 --evict_failed_followers=false --enable_leader_failure_detection=false with env {} I0504 12:39:09.809518 3497 heartbeater.cc:345] Connected to a master server at 127.0.57.254:43043 I0504 12:39:09.809736 3497 heartbeater.cc:426] Registering TS with master... I0504 12:39:09.810400 3497 heartbeater.cc:475] Master 127.0.57.254:43043 requested a full tablet report, sending... I0504 12:39:09.812389 3334 ts_manager.cc:159] Registered new tserver with Master: 1db2335bd82b489ea0f4303451133b62 (127.0.57.193:37541) I0504 12:39:09.814132 3334 master_service.cc:273] Signed X509 certificate for tserver {username='slave'} at 127.0.57.193:57933 WARNING: Logging before InitGoogleLogging() is written to STDERR W0504 12:39:09.946684 3501 flags.cc:404] Enabled unsafe flag: --enable_leader_failure_detection=false W0504 12:39:09.946972 3501 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0504 12:39:09.947075 3501 flags.cc:404] Enabled unsafe flag: --never_fsync=true W0504 12:39:09.949265 3501 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W0504 12:39:09.949400 3501 flags.cc:404] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.57.194 I0504 12:39:09.953299 3501 tablet_server_main.cc:78] Tablet server non-default flags: --enable_leader_failure_detection=false --evict_failed_followers=false --raft_heartbeat_interval_ms=50 --fs_data_dirs=/tmp/dist-test-taskHOObI0/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromoteAndDemote.1556973524314425-231/raft_consensus-itest-cluster/ts-1/data --fs_wal_dir=/tmp/dist-test-taskHOObI0/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromoteAndDemote.1556973524314425-231/raft_consensus-itest-cluster/ts-1/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.0.57.194:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskHOObI0/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromoteAndDemote.1556973524314425-231/raft_consensus-itest-cluster/ts-1/data/info.pb --webserver_interface=127.0.57.194 --webserver_port=0 --tserver_master_addrs=127.0.57.254:43043 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.0.57.194 --log_dir=/tmp/dist-test-taskHOObI0/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromoteAndDemote.1556973524314425-231/raft_consensus-itest-cluster/ts-1/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.10.0-cdh6.x-SNAPSHOT revision 9f71c857a3824d8541c4608dda2722f71515784f build type FASTDEBUG built by jenkins at 04 May 2019 04:53:16 PST on quasar-btakog-1.vpc.cloudera.com build id 321096 ASAN enabled I0504 12:39:09.960474 3501 system_ntp.cc:202] NTP initialized. Skew: 500ppm Current error: 48016us I0504 12:39:09.961107 3501 fs_manager.cc:263] Metadata directory not provided I0504 12:39:09.961169 3501 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0504 12:39:09.961468 3501 server_base.cc:433] Could not load existing FS layout: Not found: could not find a healthy instance file I0504 12:39:09.961529 3501 server_base.cc:434] Attempting to create new FS layout instead I0504 12:39:09.963893 3501 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskHOObI0/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromoteAndDemote.1556973524314425-231/raft_consensus-itest-cluster/ts-1/data/instance: uuid: "5d454de3f9224ab4ae992a414cfbdbae" format_stamp: "Formatted at 2019-05-04 12:39:09 on dist-test-slave-dist-test-slave-jj8h" I0504 12:39:09.964431 3501 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskHOObI0/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromoteAndDemote.1556973524314425-231/raft_consensus-itest-cluster/ts-1/wal/instance: uuid: "5d454de3f9224ab4ae992a414cfbdbae" format_stamp: "Formatted at 2019-05-04 12:39:09 on dist-test-slave-dist-test-slave-jj8h" I0504 12:39:09.970469 3501 fs_manager.cc:503] Time spent creating directory manager: real 0.006s user 0.002s sys 0.005s I0504 12:39:09.971523 3501 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0504 12:39:09.972093 3501 file_cache.cc:466] Constructed file cache lbm with capacity 419430 I0504 12:39:09.974820 3501 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s I0504 12:39:09.974912 3501 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskHOObI0/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromoteAndDemote.1556973524314425-231/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskHOObI0/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromoteAndDemote.1556973524314425-231/raft_consensus-itest-cluster/ts-1/wal uuid: "5d454de3f9224ab4ae992a414cfbdbae" format_stamp: "Formatted at 2019-05-04 12:39:09 on dist-test-slave-dist-test-slave-jj8h" I0504 12:39:09.975061 3501 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskHOObI0/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromoteAndDemote.1556973524314425-231/raft_consensus-itest-cluster/ts-1/wal metadata directory: /tmp/dist-test-taskHOObI0/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromoteAndDemote.1556973524314425-231/raft_consensus-itest-cluster/ts-1/wal 1 data directories: /tmp/dist-test-taskHOObI0/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromoteAndDemote.1556973524314425-231/raft_consensus-itest-cluster/ts-1/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0504 12:39:09.990264 3501 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0504 12:39:09.991984 3501 ts_tablet_manager.cc:356] Loaded tablet metadata (0 total tablets, 0 live tablets) I0504 12:39:09.992061 3501 ts_tablet_manager.cc:376] Registered 0 tablets I0504 12:39:10.016075 3501 rpc_server.cc:205] RPC server started. Bound to: 127.0.57.194:40333 I0504 12:39:10.017212 3501 webserver.cc:308] Webserver started at http://127.0.57.194:41107/ using document root and password file I0504 12:39:10.017911 3501 server_base.cc:608] Dumped server information to /tmp/dist-test-taskHOObI0/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromoteAndDemote.1556973524314425-231/raft_consensus-itest-cluster/ts-1/data/info.pb I0504 12:39:10.018288 231 external_mini_cluster.cc:995] Started /tmp/dist-test-taskHOObI0/build/asan/bin/kudu-tserver as pid 3501 I0504 12:39:10.019115 231 external_mini_cluster.cc:933] Running /tmp/dist-test-taskHOObI0/build/asan/bin/kudu-tserver /tmp/dist-test-taskHOObI0/build/asan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskHOObI0/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromoteAndDemote.1556973524314425-231/raft_consensus-itest-cluster/ts-2/wal --fs_data_dirs=/tmp/dist-test-taskHOObI0/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromoteAndDemote.1556973524314425-231/raft_consensus-itest-cluster/ts-2/data --block_manager=log --rpc_bind_addresses=127.0.57.195:0 --local_ip_for_outbound_sockets=127.0.57.195 --webserver_interface=127.0.57.195 --webserver_port=0 --tserver_master_addrs=127.0.57.254:43043 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskHOObI0/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromoteAndDemote.1556973524314425-231/raft_consensus-itest-cluster/ts-2/logs --server_dump_info_path=/tmp/dist-test-taskHOObI0/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromoteAndDemote.1556973524314425-231/raft_consensus-itest-cluster/ts-2/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_heartbeat_interval_ms=50 --evict_failed_followers=false --enable_leader_failure_detection=false with env {} I0504 12:39:10.031560 3620 heartbeater.cc:345] Connected to a master server at 127.0.57.254:43043 I0504 12:39:10.031824 3620 heartbeater.cc:426] Registering TS with master... I0504 12:39:10.032516 3620 heartbeater.cc:475] Master 127.0.57.254:43043 requested a full tablet report, sending... I0504 12:39:10.033833 3334 ts_manager.cc:159] Registered new tserver with Master: 5d454de3f9224ab4ae992a414cfbdbae (127.0.57.194:40333) I0504 12:39:10.034976 3334 master_service.cc:273] Signed X509 certificate for tserver {username='slave'} at 127.0.57.194:59767 WARNING: Logging before InitGoogleLogging() is written to STDERR W0504 12:39:10.172204 3623 flags.cc:404] Enabled unsafe flag: --enable_leader_failure_detection=false W0504 12:39:10.172462 3623 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0504 12:39:10.172550 3623 flags.cc:404] Enabled unsafe flag: --never_fsync=true W0504 12:39:10.174718 3623 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W0504 12:39:10.174867 3623 flags.cc:404] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.57.195 I0504 12:39:10.178769 3623 tablet_server_main.cc:78] Tablet server non-default flags: --enable_leader_failure_detection=false --evict_failed_followers=false --raft_heartbeat_interval_ms=50 --fs_data_dirs=/tmp/dist-test-taskHOObI0/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromoteAndDemote.1556973524314425-231/raft_consensus-itest-cluster/ts-2/data --fs_wal_dir=/tmp/dist-test-taskHOObI0/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromoteAndDemote.1556973524314425-231/raft_consensus-itest-cluster/ts-2/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.0.57.195:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskHOObI0/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromoteAndDemote.1556973524314425-231/raft_consensus-itest-cluster/ts-2/data/info.pb --webserver_interface=127.0.57.195 --webserver_port=0 --tserver_master_addrs=127.0.57.254:43043 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.0.57.195 --log_dir=/tmp/dist-test-taskHOObI0/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromoteAndDemote.1556973524314425-231/raft_consensus-itest-cluster/ts-2/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.10.0-cdh6.x-SNAPSHOT revision 9f71c857a3824d8541c4608dda2722f71515784f build type FASTDEBUG built by jenkins at 04 May 2019 04:53:16 PST on quasar-btakog-1.vpc.cloudera.com build id 321096 ASAN enabled I0504 12:39:10.186508 3623 system_ntp.cc:202] NTP initialized. Skew: 500ppm Current error: 48516us I0504 12:39:10.187165 3623 fs_manager.cc:263] Metadata directory not provided I0504 12:39:10.187224 3623 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0504 12:39:10.187547 3623 server_base.cc:433] Could not load existing FS layout: Not found: could not find a healthy instance file I0504 12:39:10.187608 3623 server_base.cc:434] Attempting to create new FS layout instead I0504 12:39:10.190029 3623 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskHOObI0/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromoteAndDemote.1556973524314425-231/raft_consensus-itest-cluster/ts-2/data/instance: uuid: "c85371c011254fa9aba0b5d644912d2c" format_stamp: "Formatted at 2019-05-04 12:39:10 on dist-test-slave-dist-test-slave-jj8h" I0504 12:39:10.190893 3623 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskHOObI0/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromoteAndDemote.1556973524314425-231/raft_consensus-itest-cluster/ts-2/wal/instance: uuid: "c85371c011254fa9aba0b5d644912d2c" format_stamp: "Formatted at 2019-05-04 12:39:10 on dist-test-slave-dist-test-slave-jj8h" I0504 12:39:10.197418 3623 fs_manager.cc:503] Time spent creating directory manager: real 0.006s user 0.001s sys 0.004s I0504 12:39:10.198463 3623 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0504 12:39:10.198997 3623 file_cache.cc:466] Constructed file cache lbm with capacity 419430 I0504 12:39:10.201725 3623 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.000s sys 0.000s I0504 12:39:10.201812 3623 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskHOObI0/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromoteAndDemote.1556973524314425-231/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskHOObI0/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromoteAndDemote.1556973524314425-231/raft_consensus-itest-cluster/ts-2/wal uuid: "c85371c011254fa9aba0b5d644912d2c" format_stamp: "Formatted at 2019-05-04 12:39:10 on dist-test-slave-dist-test-slave-jj8h" I0504 12:39:10.201978 3623 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskHOObI0/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromoteAndDemote.1556973524314425-231/raft_consensus-itest-cluster/ts-2/wal metadata directory: /tmp/dist-test-taskHOObI0/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromoteAndDemote.1556973524314425-231/raft_consensus-itest-cluster/ts-2/wal 1 data directories: /tmp/dist-test-taskHOObI0/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromoteAndDemote.1556973524314425-231/raft_consensus-itest-cluster/ts-2/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0504 12:39:10.254412 3623 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0504 12:39:10.255955 3623 ts_tablet_manager.cc:356] Loaded tablet metadata (0 total tablets, 0 live tablets) I0504 12:39:10.256033 3623 ts_tablet_manager.cc:376] Registered 0 tablets I0504 12:39:10.279664 3623 rpc_server.cc:205] RPC server started. Bound to: 127.0.57.195:35639 I0504 12:39:10.280815 3623 webserver.cc:308] Webserver started at http://127.0.57.195:33569/ using document root and password file I0504 12:39:10.281461 3623 server_base.cc:608] Dumped server information to /tmp/dist-test-taskHOObI0/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromoteAndDemote.1556973524314425-231/raft_consensus-itest-cluster/ts-2/data/info.pb I0504 12:39:10.287967 231 external_mini_cluster.cc:995] Started /tmp/dist-test-taskHOObI0/build/asan/bin/kudu-tserver as pid 3623 I0504 12:39:10.288651 231 external_mini_cluster.cc:933] Running /tmp/dist-test-taskHOObI0/build/asan/bin/kudu-tserver /tmp/dist-test-taskHOObI0/build/asan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskHOObI0/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromoteAndDemote.1556973524314425-231/raft_consensus-itest-cluster/ts-3/wal --fs_data_dirs=/tmp/dist-test-taskHOObI0/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromoteAndDemote.1556973524314425-231/raft_consensus-itest-cluster/ts-3/data --block_manager=log --rpc_bind_addresses=127.0.57.196:0 --local_ip_for_outbound_sockets=127.0.57.196 --webserver_interface=127.0.57.196 --webserver_port=0 --tserver_master_addrs=127.0.57.254:43043 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskHOObI0/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromoteAndDemote.1556973524314425-231/raft_consensus-itest-cluster/ts-3/logs --server_dump_info_path=/tmp/dist-test-taskHOObI0/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromoteAndDemote.1556973524314425-231/raft_consensus-itest-cluster/ts-3/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_heartbeat_interval_ms=50 --evict_failed_followers=false --enable_leader_failure_detection=false with env {} I0504 12:39:10.294770 3742 heartbeater.cc:345] Connected to a master server at 127.0.57.254:43043 I0504 12:39:10.294997 3742 heartbeater.cc:426] Registering TS with master... I0504 12:39:10.295541 3742 heartbeater.cc:475] Master 127.0.57.254:43043 requested a full tablet report, sending... I0504 12:39:10.296736 3334 ts_manager.cc:159] Registered new tserver with Master: c85371c011254fa9aba0b5d644912d2c (127.0.57.195:35639) I0504 12:39:10.297880 3334 master_service.cc:273] Signed X509 certificate for tserver {username='slave'} at 127.0.57.195:34037 WARNING: Logging before InitGoogleLogging() is written to STDERR W0504 12:39:10.441287 3745 flags.cc:404] Enabled unsafe flag: --enable_leader_failure_detection=false W0504 12:39:10.441548 3745 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0504 12:39:10.441640 3745 flags.cc:404] Enabled unsafe flag: --never_fsync=true W0504 12:39:10.443773 3745 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W0504 12:39:10.443914 3745 flags.cc:404] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.57.196 I0504 12:39:10.447789 3745 tablet_server_main.cc:78] Tablet server non-default flags: --enable_leader_failure_detection=false --evict_failed_followers=false --raft_heartbeat_interval_ms=50 --fs_data_dirs=/tmp/dist-test-taskHOObI0/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromoteAndDemote.1556973524314425-231/raft_consensus-itest-cluster/ts-3/data --fs_wal_dir=/tmp/dist-test-taskHOObI0/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromoteAndDemote.1556973524314425-231/raft_consensus-itest-cluster/ts-3/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.0.57.196:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskHOObI0/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromoteAndDemote.1556973524314425-231/raft_consensus-itest-cluster/ts-3/data/info.pb --webserver_interface=127.0.57.196 --webserver_port=0 --tserver_master_addrs=127.0.57.254:43043 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.0.57.196 --log_dir=/tmp/dist-test-taskHOObI0/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromoteAndDemote.1556973524314425-231/raft_consensus-itest-cluster/ts-3/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.10.0-cdh6.x-SNAPSHOT revision 9f71c857a3824d8541c4608dda2722f71515784f build type FASTDEBUG built by jenkins at 04 May 2019 04:53:16 PST on quasar-btakog-1.vpc.cloudera.com build id 321096 ASAN enabled I0504 12:39:10.454936 3745 system_ntp.cc:202] NTP initialized. Skew: 500ppm Current error: 48516us I0504 12:39:10.455556 3745 fs_manager.cc:263] Metadata directory not provided I0504 12:39:10.455623 3745 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0504 12:39:10.455916 3745 server_base.cc:433] Could not load existing FS layout: Not found: could not find a healthy instance file I0504 12:39:10.455983 3745 server_base.cc:434] Attempting to create new FS layout instead I0504 12:39:10.458148 3745 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskHOObI0/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromoteAndDemote.1556973524314425-231/raft_consensus-itest-cluster/ts-3/data/instance: uuid: "ee5cdbde7a244812acf41283ed4100b0" format_stamp: "Formatted at 2019-05-04 12:39:10 on dist-test-slave-dist-test-slave-jj8h" I0504 12:39:10.458686 3745 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskHOObI0/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromoteAndDemote.1556973524314425-231/raft_consensus-itest-cluster/ts-3/wal/instance: uuid: "ee5cdbde7a244812acf41283ed4100b0" format_stamp: "Formatted at 2019-05-04 12:39:10 on dist-test-slave-dist-test-slave-jj8h" I0504 12:39:10.464764 3745 fs_manager.cc:503] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s I0504 12:39:10.466001 3745 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0504 12:39:10.466655 3745 file_cache.cc:466] Constructed file cache lbm with capacity 419430 I0504 12:39:10.469280 3745 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.000s sys 0.000s I0504 12:39:10.469360 3745 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskHOObI0/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromoteAndDemote.1556973524314425-231/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskHOObI0/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromoteAndDemote.1556973524314425-231/raft_consensus-itest-cluster/ts-3/wal uuid: "ee5cdbde7a244812acf41283ed4100b0" format_stamp: "Formatted at 2019-05-04 12:39:10 on dist-test-slave-dist-test-slave-jj8h" I0504 12:39:10.469493 3745 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskHOObI0/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromoteAndDemote.1556973524314425-231/raft_consensus-itest-cluster/ts-3/wal metadata directory: /tmp/dist-test-taskHOObI0/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromoteAndDemote.1556973524314425-231/raft_consensus-itest-cluster/ts-3/wal 1 data directories: /tmp/dist-test-taskHOObI0/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromoteAndDemote.1556973524314425-231/raft_consensus-itest-cluster/ts-3/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0504 12:39:10.492236 3745 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0504 12:39:10.493733 3745 ts_tablet_manager.cc:356] Loaded tablet metadata (0 total tablets, 0 live tablets) I0504 12:39:10.493810 3745 ts_tablet_manager.cc:376] Registered 0 tablets I0504 12:39:10.519306 3745 rpc_server.cc:205] RPC server started. Bound to: 127.0.57.196:44497 I0504 12:39:10.520419 3745 webserver.cc:308] Webserver started at http://127.0.57.196:43777/ using document root and password file I0504 12:39:10.521019 3745 server_base.cc:608] Dumped server information to /tmp/dist-test-taskHOObI0/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromoteAndDemote.1556973524314425-231/raft_consensus-itest-cluster/ts-3/data/info.pb I0504 12:39:10.524410 231 external_mini_cluster.cc:995] Started /tmp/dist-test-taskHOObI0/build/asan/bin/kudu-tserver as pid 3745 I0504 12:39:10.535925 3864 heartbeater.cc:345] Connected to a master server at 127.0.57.254:43043 I0504 12:39:10.536160 3864 heartbeater.cc:426] Registering TS with master... I0504 12:39:10.536710 3864 heartbeater.cc:475] Master 127.0.57.254:43043 requested a full tablet report, sending... I0504 12:39:10.537948 3334 ts_manager.cc:159] Registered new tserver with Master: ee5cdbde7a244812acf41283ed4100b0 (127.0.57.196:44497) I0504 12:39:10.538928 3334 master_service.cc:273] Signed X509 certificate for tserver {username='slave'} at 127.0.57.196:57571 I0504 12:39:10.538981 231 external_mini_cluster.cc:623] 4 TS(s) registered with all masters I0504 12:39:10.550263 3334 catalog_manager.cc:1415] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46256: name: "TestTable" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 3 split_rows_range_bounds { } partition_schema { range_schema { columns { name: "key" } } } I0504 12:39:10.599808 3557 tablet_service.cc:1016] Processing CreateTablet for tablet 7f27e90fac8b4887b84d167318670d5e (table=TestTable [id=f522fd5a8b0a4d0f823f4b304ac0b265]), partition=RANGE (key) PARTITION UNBOUNDED I0504 12:39:10.600903 3557 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet 7f27e90fac8b4887b84d167318670d5e. 1 dirs total, 0 dirs full, 0 dirs failed I0504 12:39:10.600775 3679 tablet_service.cc:1016] Processing CreateTablet for tablet 7f27e90fac8b4887b84d167318670d5e (table=TestTable [id=f522fd5a8b0a4d0f823f4b304ac0b265]), partition=RANGE (key) PARTITION UNBOUNDED I0504 12:39:10.600773 3434 tablet_service.cc:1016] Processing CreateTablet for tablet 7f27e90fac8b4887b84d167318670d5e (table=TestTable [id=f522fd5a8b0a4d0f823f4b304ac0b265]), partition=RANGE (key) PARTITION UNBOUNDED I0504 12:39:10.601809 3434 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet 7f27e90fac8b4887b84d167318670d5e. 1 dirs total, 0 dirs full, 0 dirs failed I0504 12:39:10.601809 3679 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet 7f27e90fac8b4887b84d167318670d5e. 1 dirs total, 0 dirs full, 0 dirs failed I0504 12:39:10.611271 3883 tablet_bootstrap.cc:439] T 7f27e90fac8b4887b84d167318670d5e P 5d454de3f9224ab4ae992a414cfbdbae: Bootstrap starting. I0504 12:39:10.613664 3883 tablet_bootstrap.cc:590] T 7f27e90fac8b4887b84d167318670d5e P 5d454de3f9224ab4ae992a414cfbdbae: No blocks or log segments found. Creating new log. I0504 12:39:10.614754 3884 tablet_bootstrap.cc:439] T 7f27e90fac8b4887b84d167318670d5e P c85371c011254fa9aba0b5d644912d2c: Bootstrap starting. I0504 12:39:10.616755 3884 tablet_bootstrap.cc:590] T 7f27e90fac8b4887b84d167318670d5e P c85371c011254fa9aba0b5d644912d2c: No blocks or log segments found. Creating new log. I0504 12:39:10.617560 3884 log.cc:525] T 7f27e90fac8b4887b84d167318670d5e P c85371c011254fa9aba0b5d644912d2c: Log is configured to *not* fsync() on all Append() calls I0504 12:39:10.618636 3883 log.cc:525] T 7f27e90fac8b4887b84d167318670d5e P 5d454de3f9224ab4ae992a414cfbdbae: Log is configured to *not* fsync() on all Append() calls I0504 12:39:10.618966 3885 tablet_bootstrap.cc:439] T 7f27e90fac8b4887b84d167318670d5e P 1db2335bd82b489ea0f4303451133b62: Bootstrap starting. I0504 12:39:10.620810 3885 tablet_bootstrap.cc:590] T 7f27e90fac8b4887b84d167318670d5e P 1db2335bd82b489ea0f4303451133b62: No blocks or log segments found. Creating new log. I0504 12:39:10.621542 3885 log.cc:525] T 7f27e90fac8b4887b84d167318670d5e P 1db2335bd82b489ea0f4303451133b62: Log is configured to *not* fsync() on all Append() calls I0504 12:39:10.623039 3883 tablet_bootstrap.cc:439] T 7f27e90fac8b4887b84d167318670d5e P 5d454de3f9224ab4ae992a414cfbdbae: No bootstrap required, opened a new log I0504 12:39:10.623298 3883 ts_tablet_manager.cc:1053] T 7f27e90fac8b4887b84d167318670d5e P 5d454de3f9224ab4ae992a414cfbdbae: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.003s I0504 12:39:10.623296 3885 tablet_bootstrap.cc:439] T 7f27e90fac8b4887b84d167318670d5e P 1db2335bd82b489ea0f4303451133b62: No bootstrap required, opened a new log I0504 12:39:10.623292 3884 tablet_bootstrap.cc:439] T 7f27e90fac8b4887b84d167318670d5e P c85371c011254fa9aba0b5d644912d2c: No bootstrap required, opened a new log I0504 12:39:10.623531 3885 ts_tablet_manager.cc:1053] T 7f27e90fac8b4887b84d167318670d5e P 1db2335bd82b489ea0f4303451133b62: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s I0504 12:39:10.623531 3884 ts_tablet_manager.cc:1053] T 7f27e90fac8b4887b84d167318670d5e P c85371c011254fa9aba0b5d644912d2c: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.000s I0504 12:39:10.629797 3883 raft_consensus.cc:340] T 7f27e90fac8b4887b84d167318670d5e P 5d454de3f9224ab4ae992a414cfbdbae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c85371c011254fa9aba0b5d644912d2c" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 35639 } } peers { permanent_uuid: "5d454de3f9224ab4ae992a414cfbdbae" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 40333 } } peers { permanent_uuid: "1db2335bd82b489ea0f4303451133b62" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 37541 } } I0504 12:39:10.630151 3883 raft_consensus.cc:705] T 7f27e90fac8b4887b84d167318670d5e P 5d454de3f9224ab4ae992a414cfbdbae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d454de3f9224ab4ae992a414cfbdbae, State: Initialized, Role: FOLLOWER I0504 12:39:10.630497 3883 consensus_queue.cc:229] T 7f27e90fac8b4887b84d167318670d5e P 5d454de3f9224ab4ae992a414cfbdbae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c85371c011254fa9aba0b5d644912d2c" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 35639 } } peers { permanent_uuid: "5d454de3f9224ab4ae992a414cfbdbae" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 40333 } } peers { permanent_uuid: "1db2335bd82b489ea0f4303451133b62" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 37541 } } I0504 12:39:10.632210 3884 raft_consensus.cc:340] T 7f27e90fac8b4887b84d167318670d5e P c85371c011254fa9aba0b5d644912d2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c85371c011254fa9aba0b5d644912d2c" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 35639 } } peers { permanent_uuid: "5d454de3f9224ab4ae992a414cfbdbae" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 40333 } } peers { permanent_uuid: "1db2335bd82b489ea0f4303451133b62" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 37541 } } I0504 12:39:10.632627 3884 raft_consensus.cc:705] T 7f27e90fac8b4887b84d167318670d5e P c85371c011254fa9aba0b5d644912d2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c85371c011254fa9aba0b5d644912d2c, State: Initialized, Role: FOLLOWER I0504 12:39:10.632511 3885 raft_consensus.cc:340] T 7f27e90fac8b4887b84d167318670d5e P 1db2335bd82b489ea0f4303451133b62 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c85371c011254fa9aba0b5d644912d2c" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 35639 } } peers { permanent_uuid: "5d454de3f9224ab4ae992a414cfbdbae" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 40333 } } peers { permanent_uuid: "1db2335bd82b489ea0f4303451133b62" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 37541 } } I0504 12:39:10.632869 3885 raft_consensus.cc:705] T 7f27e90fac8b4887b84d167318670d5e P 1db2335bd82b489ea0f4303451133b62 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1db2335bd82b489ea0f4303451133b62, State: Initialized, Role: FOLLOWER I0504 12:39:10.632948 3884 consensus_queue.cc:229] T 7f27e90fac8b4887b84d167318670d5e P c85371c011254fa9aba0b5d644912d2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c85371c011254fa9aba0b5d644912d2c" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 35639 } } peers { permanent_uuid: "5d454de3f9224ab4ae992a414cfbdbae" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 40333 } } peers { permanent_uuid: "1db2335bd82b489ea0f4303451133b62" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 37541 } } I0504 12:39:10.633126 3885 consensus_queue.cc:229] T 7f27e90fac8b4887b84d167318670d5e P 1db2335bd82b489ea0f4303451133b62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c85371c011254fa9aba0b5d644912d2c" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 35639 } } peers { permanent_uuid: "5d454de3f9224ab4ae992a414cfbdbae" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 40333 } } peers { permanent_uuid: "1db2335bd82b489ea0f4303451133b62" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 37541 } } I0504 12:39:10.633829 3620 heartbeater.cc:467] Master 127.0.57.254:43043 was elected leader, sending a full tablet report... I0504 12:39:10.633949 3742 heartbeater.cc:467] Master 127.0.57.254:43043 was elected leader, sending a full tablet report... I0504 12:39:10.634088 3497 heartbeater.cc:467] Master 127.0.57.254:43043 was elected leader, sending a full tablet report... I0504 12:39:10.634140 3883 ts_tablet_manager.cc:1081] T 7f27e90fac8b4887b84d167318670d5e P 5d454de3f9224ab4ae992a414cfbdbae: Time spent starting tablet: real 0.011s user 0.007s sys 0.004s I0504 12:39:10.635092 3885 ts_tablet_manager.cc:1081] T 7f27e90fac8b4887b84d167318670d5e P 1db2335bd82b489ea0f4303451133b62: Time spent starting tablet: real 0.011s user 0.011s sys 0.000s I0504 12:39:10.635299 3884 ts_tablet_manager.cc:1081] T 7f27e90fac8b4887b84d167318670d5e P c85371c011254fa9aba0b5d644912d2c: Time spent starting tablet: real 0.012s user 0.006s sys 0.005s I0504 12:39:10.643723 231 external_mini_cluster.cc:623] 4 TS(s) registered with all masters I0504 12:39:10.645020 231 ts_itest-base.cc:232] Waiting for 1 tablets on tserver 1db2335bd82b489ea0f4303451133b62 to finish bootstrapping I0504 12:39:10.651185 231 ts_itest-base.cc:232] Waiting for 1 tablets on tserver 5d454de3f9224ab4ae992a414cfbdbae to finish bootstrapping I0504 12:39:10.657069 231 ts_itest-base.cc:232] Waiting for 1 tablets on tserver c85371c011254fa9aba0b5d644912d2c to finish bootstrapping I0504 12:39:10.662788 231 ts_itest-base.cc:232] Waiting for 0 tablets on tserver ee5cdbde7a244812acf41283ed4100b0 to finish bootstrapping I0504 12:39:10.675451 3454 tablet_service.cc:1403] Received Run Leader Election RPC: tablet_id: "7f27e90fac8b4887b84d167318670d5e" dest_uuid: "1db2335bd82b489ea0f4303451133b62" from {username='slave'} at 127.0.0.1:59244 I0504 12:39:10.675760 3454 raft_consensus.cc:472] T 7f27e90fac8b4887b84d167318670d5e P 1db2335bd82b489ea0f4303451133b62 [term 0 FOLLOWER]: Starting forced leader election (received explicit request) I0504 12:39:10.675927 3454 raft_consensus.cc:2886] T 7f27e90fac8b4887b84d167318670d5e P 1db2335bd82b489ea0f4303451133b62 [term 0 FOLLOWER]: Advancing to term 1 W0504 12:39:10.770045 3621 tablet.cc:1786] T 7f27e90fac8b4887b84d167318670d5e P 5d454de3f9224ab4ae992a414cfbdbae: Can't schedule compaction. Clean time has not been advanced past its initial value. W0504 12:39:10.783010 3743 tablet.cc:1786] T 7f27e90fac8b4887b84d167318670d5e P c85371c011254fa9aba0b5d644912d2c: Can't schedule compaction. Clean time has not been advanced past its initial value. W0504 12:39:10.792601 3498 tablet.cc:1786] T 7f27e90fac8b4887b84d167318670d5e P 1db2335bd82b489ea0f4303451133b62: Can't schedule compaction. Clean time has not been advanced past its initial value. I0504 12:39:11.088824 3454 raft_consensus.cc:494] T 7f27e90fac8b4887b84d167318670d5e P 1db2335bd82b489ea0f4303451133b62 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c85371c011254fa9aba0b5d644912d2c" member_type: VOTER last_known_addr { host: "127.0.57.195" port: 35639 } } peers { permanent_uuid: "5d454de3f9224ab4ae992a414cfbdbae" member_type: VOTER last_known_addr { host: "127.0.57.194" port: 40333 } } peers { permanent_uuid: "1db2335bd82b489ea0f4303451133b62" member_type: VOTER last_known_addr { host: "127.0.57.193" port: 37541 } } I0504 12:39:11.090312 3454 leader_election.cc:296] T 7f27e90fac8b4887b84d167318670d5e P 1db2335bd82b489ea0f4303451133b62 [CANDIDATE]: Term 1 election: Requested vote from peers c85371c011254fa9aba0b5d644912d2c (127.0.57.195:35639), 5d454de3f9224ab4ae992a414cfbdbae (127.0.57.194:40333) I0504 12:39:11.098052 3699 tablet_service.cc:1286] Received RequestConsensusVote() RPC: tablet_id: "7f27e90fac8b4887b84d167318670d5e" candidate_uuid: "1db2335bd82b489ea0f4303451133b62" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "c85371c011254fa9aba0b5d644912d2c" I0504 12:39:11.098013 3577 tablet_service.cc:1286] Received RequestConsensusVote() RPC: tablet_id: "7f27e90fac8b4887b84d167318670d5e" candidate_uuid: "1db2335bd82b489ea0f4303451133b62" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "5d454de3f9224ab4ae992a414cfbdbae" I0504 12:39:11.098351 3577 raft_consensus.cc:2886] T 7f27e90fac8b4887b84d167318670d5e P 5d454de3f9224ab4ae992a414cfbdbae [term 0 FOLLOWER]: Advancing to term 1 I0504 12:39:11.098345 3699 raft_consensus.cc:2886] T 7f27e90fac8b4887b84d167318670d5e P c85371c011254fa9aba0b5d644912d2c [term 0 FOLLOWER]: Advancing to term 1 W0504 12:39:11.261185 3385 leader_election.cc:341] T 7f27e90fac8b4887b84d167318670d5e P 1db2335bd82b489ea0f4303451133b62 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer c85371c011254fa9aba0b5d644912d2c (127.0.57.195:35639): Timed out: RequestConsensusVote RPC to 127.0.57.195:35639 timed out after 0.170s (SENT) W0504 12:39:11.261538 3387 leader_election.cc:341] T 7f27e90fac8b4887b84d167318670d5e P 1db2335bd82b489ea0f4303451133b62 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 5d454de3f9224ab4ae992a414cfbdbae (127.0.57.194:40333): Timed out: RequestConsensusVote RPC to 127.0.57.194:40333 timed out after 0.170s (SENT) I0504 12:39:11.261698 3387 leader_election.cc:310] T 7f27e90fac8b4887b84d167318670d5e P 1db2335bd82b489ea0f4303451133b62 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1db2335bd82b489ea0f4303451133b62; no voters: 5d454de3f9224ab4ae992a414cfbdbae, c85371c011254fa9aba0b5d644912d2c I0504 12:39:11.262462 3896 raft_consensus.cc:2588] T 7f27e90fac8b4887b84d167318670d5e P 1db2335bd82b489ea0f4303451133b62 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority I0504 12:39:11.540894 3864 heartbeater.cc:467] Master 127.0.57.254:43043 was elected leader, sending a full tablet report... W0504 12:39:11.599707 3577 consensus_meta.cc:220] T 7f27e90fac8b4887b84d167318670d5e P 5d454de3f9224ab4ae992a414cfbdbae: Time spent flushing consensus metadata: real 0.501s user 0.003s sys 0.000s W0504 12:39:11.599707 3699 consensus_meta.cc:220] T 7f27e90fac8b4887b84d167318670d5e P c85371c011254fa9aba0b5d644912d2c: Time spent flushing consensus metadata: real 0.501s user 0.004s sys 0.000s I0504 12:39:11.599836 3699 raft_consensus.cc:2346] T 7f27e90fac8b4887b84d167318670d5e P c85371c011254fa9aba0b5d644912d2c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1db2335bd82b489ea0f4303451133b62 in term 1. I0504 12:39:11.599838 3577 raft_consensus.cc:2346] T 7f27e90fac8b4887b84d167318670d5e P 5d454de3f9224ab4ae992a414cfbdbae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1db2335bd82b489ea0f4303451133b62 in term 1. W0504 12:39:11.600097 3699 rpcz_store.cc:253] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.57.193:38713 (request call id 1) took 502 ms (502 ms). Client timeout 170 ms (170 ms) W0504 12:39:11.600124 3577 rpcz_store.cc:253] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.57.193:39809 (request call id 1) took 502 ms (502 ms). Client timeout 170 ms (170 ms) W0504 12:39:11.600728 3699 rpcz_store.cc:259] Trace: 0504 12:39:11.097610 (+ 0us) service_pool.cc:162] Inserting onto call queue 0504 12:39:11.097680 (+ 70us) service_pool.cc:221] Handling call 0504 12:39:11.600036 (+502356us) inbound_call.cc:162] Queueing success response Metrics: {} W0504 12:39:11.600728 3577 rpcz_store.cc:259] Trace: 0504 12:39:11.097443 (+ 0us) service_pool.cc:162] Inserting onto call queue 0504 12:39:11.097552 (+ 109us) service_pool.cc:221] Handling call 0504 12:39:11.600035 (+502483us) inbound_call.cc:162] Queueing success response Metrics: {} I0504 12:41:09.599308 3323 maintenance_manager.cc:306] P 3744a79ca8264f1587a6daa834a73948: Scheduling FlushMRSOp(00000000000000000000000000000000): perf score=0.033371 I0504 12:41:09.626622 3322 maintenance_manager.cc:501] P 3744a79ca8264f1587a6daa834a73948: FlushMRSOp(00000000000000000000000000000000) complete. Timing: real 0.026s user 0.026s sys 0.000s Metrics: {"bytes_written":6723,"cfile_init":1,"data dirs.queue_time_us":1089,"data dirs.run_cpu_time_us":298,"data dirs.run_wall_time_us":743,"drs_written":1,"lbm_read_time_us":70,"lbm_reads_lt_1ms":4,"lbm_write_time_us":478,"lbm_writes_lt_1ms":27,"rows_written":4,"thread_start_us":1335,"threads_started":2,"wal-append.queue_time_us":338} I0504 12:41:09.627954 3323 maintenance_manager.cc:306] P 3744a79ca8264f1587a6daa834a73948: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 479 bytes on disk I0504 12:41:09.628757 3322 maintenance_manager.cc:501] P 3744a79ca8264f1587a6daa834a73948: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":70,"lbm_reads_lt_1ms":4} /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/raft_consensus_nonvoter-itest.cc:859: Failure Failed Bad status: Timed out: Replica TabletServer: 1db2335bd82b489ea0f4303451133b62, Rpc address: host: "127.0.57.193" port: 37541 is not leader after waiting for 120.000s: Not found: Error connecting to replica: Timed out: GetConsensusState RPC to 127.0.57.193:37541 timed out after -0.037s (ON_OUTBOUND_QUEUE) I0504 12:41:11.134759 231 external_mini_cluster.cc:1172] Attempting to check leaks for /tmp/dist-test-taskHOObI0/build/asan/bin/kudu-tserver pid 3368 I0504 12:41:11.142941 3494 generic_service.cc:171] Checking for leaks (request via RPC) I0504 12:41:11.210763 231 external_mini_cluster.cc:1131] Killing /tmp/dist-test-taskHOObI0/build/asan/bin/kudu-tserver with pid 3368 I0504 12:41:11.223901 231 external_mini_cluster.cc:1172] Attempting to check leaks for /tmp/dist-test-taskHOObI0/build/asan/bin/kudu-tserver pid 3501 I0504 12:41:11.229043 3617 generic_service.cc:171] Checking for leaks (request via RPC) I0504 12:41:11.282596 231 external_mini_cluster.cc:1131] Killing /tmp/dist-test-taskHOObI0/build/asan/bin/kudu-tserver with pid 3501 I0504 12:41:11.296147 231 external_mini_cluster.cc:1172] Attempting to check leaks for /tmp/dist-test-taskHOObI0/build/asan/bin/kudu-tserver pid 3623 I0504 12:41:11.301869 3739 generic_service.cc:171] Checking for leaks (request via RPC) I0504 12:41:11.363757 231 external_mini_cluster.cc:1131] Killing /tmp/dist-test-taskHOObI0/build/asan/bin/kudu-tserver with pid 3623 I0504 12:41:11.376716 231 external_mini_cluster.cc:1172] Attempting to check leaks for /tmp/dist-test-taskHOObI0/build/asan/bin/kudu-tserver pid 3745 I0504 12:41:11.382122 3861 generic_service.cc:171] Checking for leaks (request via RPC) I0504 12:41:11.435705 231 external_mini_cluster.cc:1131] Killing /tmp/dist-test-taskHOObI0/build/asan/bin/kudu-tserver with pid 3745 I0504 12:41:11.447526 231 external_mini_cluster.cc:1172] Attempting to check leaks for /tmp/dist-test-taskHOObI0/build/asan/bin/kudu-master pid 3308 I0504 12:41:11.453375 3364 generic_service.cc:171] Checking for leaks (request via RPC) I0504 12:41:11.552709 231 external_mini_cluster.cc:1131] Killing /tmp/dist-test-taskHOObI0/build/asan/bin/kudu-master with pid 3308 I0504 12:41:11.572476 231 test_util.cc:135] ----------------------------------------------- I0504 12:41:11.572540 231 test_util.cc:136] Had fatal failures, leaving test files at /tmp/dist-test-taskHOObI0/test-tmp/raft_consensus_nonvoter-itest.0.RaftConsensusNonVoterITest.PromoteAndDemote.1556973524314425-231 [ FAILED ] RaftConsensusNonVoterITest.PromoteAndDemote (122426 ms)