[==========] Running 1 test from 1 test case. [----------] Global test environment set-up. [----------] 1 test from RaftConsensusStressITest [ RUN ] RaftConsensusStressITest.RemoveReplaceInCycle WARNING: Logging before InitGoogleLogging() is written to STDERR I1017 04:27:17.724267 31197 test_util.cc:209] Using random seed: -202662838 I1017 04:27:17.742168 31197 ts_itest-base.cc:113] Starting cluster with: I1017 04:27:17.742223 31197 ts_itest-base.cc:114] -------------- I1017 04:27:17.742250 31197 ts_itest-base.cc:115] 6 tablet servers I1017 04:27:17.742278 31197 ts_itest-base.cc:116] 3 replicas per TS I1017 04:27:17.742312 31197 ts_itest-base.cc:117] -------------- I1017 04:27:17.744487 31197 external_mini_cluster.cc:845] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-master /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-master --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1539775637668737-31197/raft_consensus-itest-cluster/master-0/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1539775637668737-31197/raft_consensus-itest-cluster/master-0/data --block_manager=log --webserver_interface=localhost --ipki_ca_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.30.119.126:56502 --webserver_interface=127.30.119.126 --webserver_port=0 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1539775637668737-31197/raft_consensus-itest-cluster/master-0/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1539775637668737-31197/raft_consensus-itest-cluster/master-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --minloglevel=2 --raft_prepare_replacement_before_eviction=true --max_create_tablets_per_ts=6 --rpc_reuseport=true with env {} I1017 04:27:18.067037 31197 external_mini_cluster.cc:907] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-master as pid 31207 I1017 04:27:18.068336 31197 external_mini_cluster.cc:845] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1539775637668737-31197/raft_consensus-itest-cluster/ts-0/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1539775637668737-31197/raft_consensus-itest-cluster/ts-0/data --block_manager=log --rpc_bind_addresses=127.30.119.65:0 --local_ip_for_outbound_sockets=127.30.119.65 --webserver_interface=127.30.119.65 --webserver_port=0 --tserver_master_addrs=127.30.119.126:56502 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1539775637668737-31197/raft_consensus-itest-cluster/ts-0/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1539775637668737-31197/raft_consensus-itest-cluster/ts-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --minloglevel=2 --tablet_copy_early_session_timeout_prob=0.125 --raft_prepare_replacement_before_eviction=true --follower_unavailable_considered_failed_sec=2 --consensus_rpc_timeout_ms=2000 --heartbeat_interval_ms=500 --raft_heartbeat_interval_ms=200 with env {} I1017 04:27:18.452762 31197 external_mini_cluster.cc:907] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver as pid 31267 I1017 04:27:18.453758 31197 external_mini_cluster.cc:845] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1539775637668737-31197/raft_consensus-itest-cluster/ts-1/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1539775637668737-31197/raft_consensus-itest-cluster/ts-1/data --block_manager=log --rpc_bind_addresses=127.30.119.66:0 --local_ip_for_outbound_sockets=127.30.119.66 --webserver_interface=127.30.119.66 --webserver_port=0 --tserver_master_addrs=127.30.119.126:56502 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1539775637668737-31197/raft_consensus-itest-cluster/ts-1/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1539775637668737-31197/raft_consensus-itest-cluster/ts-1/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --minloglevel=2 --tablet_copy_early_session_timeout_prob=0.125 --raft_prepare_replacement_before_eviction=true --follower_unavailable_considered_failed_sec=2 --consensus_rpc_timeout_ms=2000 --heartbeat_interval_ms=500 --raft_heartbeat_interval_ms=200 with env {} I1017 04:27:18.820813 31197 external_mini_cluster.cc:907] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver as pid 31400 I1017 04:27:18.821516 31197 external_mini_cluster.cc:845] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1539775637668737-31197/raft_consensus-itest-cluster/ts-2/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1539775637668737-31197/raft_consensus-itest-cluster/ts-2/data --block_manager=log --rpc_bind_addresses=127.30.119.67:0 --local_ip_for_outbound_sockets=127.30.119.67 --webserver_interface=127.30.119.67 --webserver_port=0 --tserver_master_addrs=127.30.119.126:56502 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1539775637668737-31197/raft_consensus-itest-cluster/ts-2/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1539775637668737-31197/raft_consensus-itest-cluster/ts-2/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --minloglevel=2 --tablet_copy_early_session_timeout_prob=0.125 --raft_prepare_replacement_before_eviction=true --follower_unavailable_considered_failed_sec=2 --consensus_rpc_timeout_ms=2000 --heartbeat_interval_ms=500 --raft_heartbeat_interval_ms=200 with env {} I1017 04:27:19.164304 31197 external_mini_cluster.cc:907] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver as pid 31522 I1017 04:27:19.165051 31197 external_mini_cluster.cc:845] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1539775637668737-31197/raft_consensus-itest-cluster/ts-3/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1539775637668737-31197/raft_consensus-itest-cluster/ts-3/data --block_manager=log --rpc_bind_addresses=127.30.119.68:0 --local_ip_for_outbound_sockets=127.30.119.68 --webserver_interface=127.30.119.68 --webserver_port=0 --tserver_master_addrs=127.30.119.126:56502 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1539775637668737-31197/raft_consensus-itest-cluster/ts-3/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1539775637668737-31197/raft_consensus-itest-cluster/ts-3/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --minloglevel=2 --tablet_copy_early_session_timeout_prob=0.125 --raft_prepare_replacement_before_eviction=true --follower_unavailable_considered_failed_sec=2 --consensus_rpc_timeout_ms=2000 --heartbeat_interval_ms=500 --raft_heartbeat_interval_ms=200 with env {} I1017 04:27:19.498550 31197 external_mini_cluster.cc:907] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver as pid 31644 I1017 04:27:19.499408 31197 external_mini_cluster.cc:845] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1539775637668737-31197/raft_consensus-itest-cluster/ts-4/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1539775637668737-31197/raft_consensus-itest-cluster/ts-4/data --block_manager=log --rpc_bind_addresses=127.30.119.69:0 --local_ip_for_outbound_sockets=127.30.119.69 --webserver_interface=127.30.119.69 --webserver_port=0 --tserver_master_addrs=127.30.119.126:56502 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1539775637668737-31197/raft_consensus-itest-cluster/ts-4/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1539775637668737-31197/raft_consensus-itest-cluster/ts-4/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --minloglevel=2 --tablet_copy_early_session_timeout_prob=0.125 --raft_prepare_replacement_before_eviction=true --follower_unavailable_considered_failed_sec=2 --consensus_rpc_timeout_ms=2000 --heartbeat_interval_ms=500 --raft_heartbeat_interval_ms=200 with env {} I1017 04:27:19.853091 31197 external_mini_cluster.cc:907] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver as pid 31766 I1017 04:27:19.853885 31197 external_mini_cluster.cc:845] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1539775637668737-31197/raft_consensus-itest-cluster/ts-5/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1539775637668737-31197/raft_consensus-itest-cluster/ts-5/data --block_manager=log --rpc_bind_addresses=127.30.119.70:0 --local_ip_for_outbound_sockets=127.30.119.70 --webserver_interface=127.30.119.70 --webserver_port=0 --tserver_master_addrs=127.30.119.126:56502 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1539775637668737-31197/raft_consensus-itest-cluster/ts-5/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1539775637668737-31197/raft_consensus-itest-cluster/ts-5/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --minloglevel=2 --tablet_copy_early_session_timeout_prob=0.125 --raft_prepare_replacement_before_eviction=true --follower_unavailable_considered_failed_sec=2 --consensus_rpc_timeout_ms=2000 --heartbeat_interval_ms=500 --raft_heartbeat_interval_ms=200 with env {} I1017 04:27:20.239334 31197 external_mini_cluster.cc:907] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver as pid 31888 I1017 04:27:20.259491 31197 external_mini_cluster.cc:535] 6 TS(s) registered with all masters I1017 04:27:20.449523 31197 external_mini_cluster.cc:535] 6 TS(s) registered with all masters I1017 04:27:20.450961 31197 ts_itest-base.cc:233] Waiting for 1 tablets on tserver ca720a15d16d4077aa7ae6218d6fbce5 to finish bootstrapping I1017 04:27:20.456154 31197 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 490555426e0148be8a38ccf90bedb317 to finish bootstrapping I1017 04:27:20.463680 31197 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 167e75d199c246a0839f72a87f6b3416 to finish bootstrapping I1017 04:27:20.470379 31197 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 803373f5e35f4a61b5a983ea8b66edca to finish bootstrapping I1017 04:27:20.475896 31197 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 88847ecc753c428b8c75790b08f9e33f to finish bootstrapping I1017 04:27:20.482457 31197 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 3ab543a6ac3449a8bdd5c44a04ea5ee6 to finish bootstrapping I1017 04:27:20.487534 31197 test_util.cc:209] Using random seed: -199899554 E1017 04:27:31.265014 32431 consensus_queue.cc:699] T ce185e91fbc5412fa8bfb9c86df7711b P 803373f5e35f4a61b5a983ea8b66edca [LEADER]: Error trying to read ahead of the log while preparing peer request: Incomplete: Op with index 42 is ahead of the local log (next sequential op: 40). Destination peer: Peer: permanent_uuid: "ca720a15d16d4077aa7ae6218d6fbce5" member_type: VOTER last_known_addr { host: "127.30.119.65" port: 50109 }, Status: INVALID_TERM, Last received: 2.42, Next index: 43, Last known committed idx: 42, Time since last communication: 0.035s W1017 04:27:35.238559 32574 negotiation.cc:313] Failed RPC negotiation. Trace: 1017 04:27:32.236586 (+ 0us) reactor.cc:583] Submitting negotiation task for client connection to 127.30.119.65:50109 1017 04:27:32.238951 (+ 2365us) negotiation.cc:98] Waiting for socket to connect 1017 04:27:32.238963 (+ 12us) client_negotiation.cc:160] Beginning negotiation 1017 04:27:32.239050 (+ 87us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 1017 04:27:35.236465 (+2997415us) negotiation.cc:304] Negotiation complete: Timed out: Client connection negotiation failed: client connection to 127.30.119.65:50109 Metrics: {"client-negotiator.queue_time_us":2224} Master Summary UUID | Address | Status ----------------------------------+----------------------+--------- 0461054002fe43ac852199cb3befad20 | 127.30.119.126:56502 | HEALTHY Flag | Value | Tags | Master ------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+------------------------- ipki_ca_key_size | 1024 | experimental | all 1 server(s) checked ipki_server_key_size | 1024 | experimental | all 1 server(s) checked never_fsync | true | unsafe,advanced | all 1 server(s) checked raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked rpc_reuseport | true | experimental | all 1 server(s) checked rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked server_dump_info_format | pb | hidden | all 1 server(s) checked server_dump_info_path | /data/somelongdirectorytoavoidrpathissues/src/kudutest/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1539775637668737-31197/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked Tablet Server Summary UUID | Address | Status ----------------------------------+---------------------+------------- 167e75d199c246a0839f72a87f6b3416 | 127.30.119.67:58765 | HEALTHY 3ab543a6ac3449a8bdd5c44a04ea5ee6 | 127.30.119.70:44157 | HEALTHY 490555426e0148be8a38ccf90bedb317 | 127.30.119.66:46821 | HEALTHY 803373f5e35f4a61b5a983ea8b66edca | 127.30.119.68:51058 | HEALTHY 88847ecc753c428b8c75790b08f9e33f | 127.30.119.69:41187 | HEALTHY ca720a15d16d4077aa7ae6218d6fbce5 | 127.30.119.65:50109 | UNAVAILABLE Error from 127.30.119.65:50109: Timed out: could not get status from server: Client connection negotiation failed: client connection to 127.30.119.65:50109 (UNAVAILABLE) Flag | Value | Tags | Tablet Server ------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+------------------------- ipki_server_key_size | 1024 | experimental | all 6 server(s) checked local_ip_for_outbound_sockets | 127.30.119.65 | experimental | 127.30.119.65:50109 local_ip_for_outbound_sockets | 127.30.119.66 | experimental | 127.30.119.66:46821 local_ip_for_outbound_sockets | 127.30.119.67 | experimental | 127.30.119.67:58765 local_ip_for_outbound_sockets | 127.30.119.68 | experimental | 127.30.119.68:51058 local_ip_for_outbound_sockets | 127.30.119.69 | experimental | 127.30.119.69:41187 local_ip_for_outbound_sockets | 127.30.119.70 | experimental | 127.30.119.70:44157 never_fsync | true | unsafe,advanced | all 6 server(s) checked raft_prepare_replacement_before_eviction | true | experimental,advanced | all 6 server(s) checked rpc_server_allow_ephemeral_ports | true | unsafe | all 6 server(s) checked server_dump_info_format | pb | hidden | all 6 server(s) checked server_dump_info_path | /data/somelongdirectorytoavoidrpathissues/src/kudutest/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1539775637668737-31197/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.30.119.65:50109 server_dump_info_path | /data/somelongdirectorytoavoidrpathissues/src/kudutest/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1539775637668737-31197/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.30.119.66:46821 server_dump_info_path | /data/somelongdirectorytoavoidrpathissues/src/kudutest/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1539775637668737-31197/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.30.119.67:58765 server_dump_info_path | /data/somelongdirectorytoavoidrpathissues/src/kudutest/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1539775637668737-31197/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.30.119.68:51058 server_dump_info_path | /data/somelongdirectorytoavoidrpathissues/src/kudutest/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1539775637668737-31197/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.30.119.69:41187 server_dump_info_path | /data/somelongdirectorytoavoidrpathissues/src/kudutest/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1539775637668737-31197/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.30.119.70:44157 tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | all 6 server(s) checked Version Summary Version | Servers ----------------+-------------------------------------------------------------------------------------------------------------- 1.9.0-SNAPSHOT | master@127.30.119.126:56502, tserver@127.30.119.67:58765, tserver@127.30.119.70:44157, and 3 other server(s) Tablet 53b982eb47024b95984ad8754627357a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING 88847ecc753c428b8c75790b08f9e33f (127.30.119.69:41187): RUNNING 167e75d199c246a0839f72a87f6b3416 (127.30.119.67:58765): RUNNING [LEADER] ca720a15d16d4077aa7ae6218d6fbce5 (127.30.119.65:50109): TS unavailable All reported replicas are: A = 88847ecc753c428b8c75790b08f9e33f B = 167e75d199c246a0839f72a87f6b3416 C = ca720a15d16d4077aa7ae6218d6fbce5 The consensus matrix is: Config source | Replicas | Current term | Config index | Committed? ---------------+------------------------+--------------+--------------+------------ master | A B* C | | | Yes A | A B* C | 1 | -1 | Yes B | A B* C | 1 | -1 | Yes C | [config not available] | | | Tablet c9a30cf14a314d718ce60f43b1fa34f9 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING 167e75d199c246a0839f72a87f6b3416 (127.30.119.67:58765): RUNNING 3ab543a6ac3449a8bdd5c44a04ea5ee6 (127.30.119.70:44157): RUNNING [LEADER] ca720a15d16d4077aa7ae6218d6fbce5 (127.30.119.65:50109): TS unavailable All reported replicas are: A = 167e75d199c246a0839f72a87f6b3416 B = 3ab543a6ac3449a8bdd5c44a04ea5ee6 C = ca720a15d16d4077aa7ae6218d6fbce5 The consensus matrix is: Config source | Replicas | Current term | Config index | Committed? ---------------+------------------------+--------------+--------------+------------ master | A B* C | | | Yes A | A B* C | 1 | -1 | Yes B | A B* C | 1 | -1 | Yes C | [config not available] | | | Tablet 9e92fd7d71aa4588aded2ecdd67095cb of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING 490555426e0148be8a38ccf90bedb317 (127.30.119.66:46821): RUNNING [LEADER] ca720a15d16d4077aa7ae6218d6fbce5 (127.30.119.65:50109): TS unavailable 167e75d199c246a0839f72a87f6b3416 (127.30.119.67:58765): RUNNING All reported replicas are: A = 490555426e0148be8a38ccf90bedb317 B = ca720a15d16d4077aa7ae6218d6fbce5 C = 167e75d199c246a0839f72a87f6b3416 The consensus matrix is: Config source | Replicas | Current term | Config index | Committed? ---------------+------------------------+--------------+--------------+------------ master | A* B C | | | Yes A | A* B C | 1 | -1 | Yes B | [config not available] | | | C | A* B C | 1 | -1 | Yes Tablet ce185e91fbc5412fa8bfb9c86df7711b of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING 490555426e0148be8a38ccf90bedb317 (127.30.119.66:46821): RUNNING [LEADER] ca720a15d16d4077aa7ae6218d6fbce5 (127.30.119.65:50109): TS unavailable 3ab543a6ac3449a8bdd5c44a04ea5ee6 (127.30.119.70:44157): RUNNING All reported replicas are: A = 490555426e0148be8a38ccf90bedb317 B = ca720a15d16d4077aa7ae6218d6fbce5 C = 3ab543a6ac3449a8bdd5c44a04ea5ee6 The consensus matrix is: Config source | Replicas | Current term | Config index | Committed? ---------------+------------------------+--------------+--------------+------------ master | A* B C | | | Yes A | A* B C | 2 | 42 | Yes B | [config not available] | | | C | A* B C | 2 | 42 | Yes Tablet f430bbb1fef44e65b387432c037df4dd of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING 88847ecc753c428b8c75790b08f9e33f (127.30.119.69:41187): RUNNING [LEADER] 167e75d199c246a0839f72a87f6b3416 (127.30.119.67:58765): RUNNING ca720a15d16d4077aa7ae6218d6fbce5 (127.30.119.65:50109): TS unavailable All reported replicas are: A = 88847ecc753c428b8c75790b08f9e33f B = 167e75d199c246a0839f72a87f6b3416 C = ca720a15d16d4077aa7ae6218d6fbce5 The consensus matrix is: Config source | Replicas | Current term | Config index | Committed? ---------------+------------------------+--------------+--------------+------------ master | A* B C | | | Yes A | A* B C | 1 | -1 | Yes B | A* B C | 1 | -1 | Yes C | [config not available] | | | Tablet bed15e9cdc6d44f69eedb4378cd2ea0f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING 490555426e0148be8a38ccf90bedb317 (127.30.119.66:46821): RUNNING 88847ecc753c428b8c75790b08f9e33f (127.30.119.69:41187): RUNNING [LEADER] ca720a15d16d4077aa7ae6218d6fbce5 (127.30.119.65:50109): TS unavailable All reported replicas are: A = 490555426e0148be8a38ccf90bedb317 B = 88847ecc753c428b8c75790b08f9e33f C = ca720a15d16d4077aa7ae6218d6fbce5 The consensus matrix is: Config source | Replicas | Current term | Config index | Committed? ---------------+------------------------+--------------+--------------+------------ master | A B* C | | | Yes A | A B* C | 2 | 46 | Yes B | A B* C | 2 | 46 | Yes C | [config not available] | | | Tablet 2932a59244e646d6a27beb9786b3e9b7 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING 88847ecc753c428b8c75790b08f9e33f (127.30.119.69:41187): RUNNING [LEADER] ca720a15d16d4077aa7ae6218d6fbce5 (127.30.119.65:50109): TS unavailable 167e75d199c246a0839f72a87f6b3416 (127.30.119.67:58765): RUNNING All reported replicas are: A = 88847ecc753c428b8c75790b08f9e33f B = ca720a15d16d4077aa7ae6218d6fbce5 C = 167e75d199c246a0839f72a87f6b3416 The consensus matrix is: Config source | Replicas | Current term | Config index | Committed? ---------------+------------------------+--------------+--------------+------------ master | A* B C | | | Yes A | A* B C | 1 | -1 | Yes B | [config not available] | | | C | A* B C | 1 | -1 | Yes Tablet 6d4d96968edc4b73a3b0de435c985e29 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING 3ab543a6ac3449a8bdd5c44a04ea5ee6 (127.30.119.70:44157): RUNNING [LEADER] ca720a15d16d4077aa7ae6218d6fbce5 (127.30.119.65:50109): TS unavailable 167e75d199c246a0839f72a87f6b3416 (127.30.119.67:58765): RUNNING All reported replicas are: A = 3ab543a6ac3449a8bdd5c44a04ea5ee6 B = ca720a15d16d4077aa7ae6218d6fbce5 C = 167e75d199c246a0839f72a87f6b3416 The consensus matrix is: Config source | Replicas | Current term | Config index | Committed? ---------------+------------------------+--------------+--------------+------------ master | A* B C | | | Yes A | A* B C | 2 | 41 | Yes B | [config not available] | | | C | A* B C | 2 | 41 | Yes Tablet 8b03926386a44d589f148dbeb73e17ca of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING 3ab543a6ac3449a8bdd5c44a04ea5ee6 (127.30.119.70:44157): RUNNING ca720a15d16d4077aa7ae6218d6fbce5 (127.30.119.65:50109): TS unavailable 490555426e0148be8a38ccf90bedb317 (127.30.119.66:46821): RUNNING [LEADER] All reported replicas are: A = 3ab543a6ac3449a8bdd5c44a04ea5ee6 B = ca720a15d16d4077aa7ae6218d6fbce5 C = 490555426e0148be8a38ccf90bedb317 The consensus matrix is: Config source | Replicas | Current term | Config index | Committed? ---------------+------------------------+--------------+--------------+------------ master | A B C* | | | Yes A | A B C | 4 | 43 | Yes B | [config not available] | | | C | A B C* | 4 | 43 | Yes Tablet 4cfb1f5eca244d988010eccdf19b6c10 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING ca720a15d16d4077aa7ae6218d6fbce5 (127.30.119.65:50109): TS unavailable 167e75d199c246a0839f72a87f6b3416 (127.30.119.67:58765): RUNNING 490555426e0148be8a38ccf90bedb317 (127.30.119.66:46821): RUNNING [LEADER] All reported replicas are: A = ca720a15d16d4077aa7ae6218d6fbce5 B = 167e75d199c246a0839f72a87f6b3416 C = 490555426e0148be8a38ccf90bedb317 The consensus matrix is: Config source | Replicas | Current term | Config index | Committed? ---------------+------------------------+--------------+--------------+------------ master | A B C* | | | Yes A | [config not available] | | | B | A B C* | 1 | -1 | Yes C | A B C* | 1 | -1 | Yes Tablet 31fde0e543f84efeb62cf97bdc602900 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING 3ab543a6ac3449a8bdd5c44a04ea5ee6 (127.30.119.70:44157): RUNNING ca720a15d16d4077aa7ae6218d6fbce5 (127.30.119.65:50109): TS unavailable 490555426e0148be8a38ccf90bedb317 (127.30.119.66:46821): RUNNING [LEADER] All reported replicas are: A = 3ab543a6ac3449a8bdd5c44a04ea5ee6 B = ca720a15d16d4077aa7ae6218d6fbce5 C = 490555426e0148be8a38ccf90bedb317 The consensus matrix is: Config source | Replicas | Current term | Config index | Committed? ---------------+------------------------+--------------+--------------+------------ master | A B C* | | | Yes A | A B C | 2 | 44 | Yes B | [config not available] | | | C | A B C* | 2 | 44 | Yes Tablet ca302959d1d94e20af56964a27aba4a4 of table 'TestTable' is under-replicated: 1 replica(s) not RUNNING ca720a15d16d4077aa7ae6218d6fbce5 (127.30.119.65:50109): TS unavailable 3ab543a6ac3449a8bdd5c44a04ea5ee6 (127.30.119.70:44157): RUNNING [LEADER] 88847ecc753c428b8c75790b08f9e33f (127.30.119.69:41187): RUNNING All reported replicas are: A = ca720a15d16d4077aa7ae6218d6fbce5 B = 3ab543a6ac3449a8bdd5c44a04ea5ee6 C = 88847ecc753c428b8c75790b08f9e33f The consensus matrix is: Config source | Replicas | Current term | Config index | Committed? ---------------+------------------------+--------------+--------------+------------ master | A B* C | | | Yes A | [config not available] | | | B | A B* C | 2 | 4 | Yes C | A B C | 2 | 4 | Yes Summary by table Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable ----------------------+----+------------------+---------------+---------+------------+------------------+------------- RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 20 | 9 | 0 | 11 | 0 TestTable | 3 | UNDER_REPLICATED | 1 | 0 | 0 | 1 | 0 | Total Count ----------------+------------- Masters | 1 Tablet Servers | 6 Tables | 2 Tablets | 21 Replicas | 63 ================== Warnings: ================== Some masters have unsafe, experimental, or hidden flags set Some tablet servers have unsafe, experimental, or hidden flags set ================== Errors: ================== Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors Corruption: table consistency check error: 2 out of 2 table(s) are not healthy FAILED W1017 04:27:41.463698 365 negotiation.cc:313] Failed RPC negotiation. Trace: 1017 04:27:38.463435 (+ 0us) reactor.cc:583] Submitting negotiation task for client connection to 127.30.119.68:51058 1017 04:27:38.471434 (+ 7999us) negotiation.cc:98] Waiting for socket to connect 1017 04:27:38.471446 (+ 12us) client_negotiation.cc:160] Beginning negotiation 1017 04:27:38.471518 (+ 72us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 1017 04:27:41.463364 (+2991846us) negotiation.cc:304] Negotiation complete: Timed out: Client connection negotiation failed: client connection to 127.30.119.68:51058 Metrics: {"client-negotiator.queue_time_us":7897,"thread_start_us":11889,"threads_started":1} Master Summary UUID | Address | Status ----------------------------------+----------------------+--------- 0461054002fe43ac852199cb3befad20 | 127.30.119.126:56502 | HEALTHY Flag | Value | Tags | Master ------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+------------------------- ipki_ca_key_size | 1024 | experimental | all 1 server(s) checked ipki_server_key_size | 1024 | experimental | all 1 server(s) checked never_fsync | true | unsafe,advanced | all 1 server(s) checked raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked rpc_reuseport | true | experimental | all 1 server(s) checked rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked server_dump_info_format | pb | hidden | all 1 server(s) checked server_dump_info_path | /data/somelongdirectorytoavoidrpathissues/src/kudutest/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1539775637668737-31197/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked Tablet Server Summary UUID | Address | Status ----------------------------------+---------------------+------------- 167e75d199c246a0839f72a87f6b3416 | 127.30.119.67:58765 | HEALTHY 3ab543a6ac3449a8bdd5c44a04ea5ee6 | 127.30.119.70:44157 | HEALTHY 490555426e0148be8a38ccf90bedb317 | 127.30.119.66:46821 | HEALTHY 88847ecc753c428b8c75790b08f9e33f | 127.30.119.69:41187 | HEALTHY ca720a15d16d4077aa7ae6218d6fbce5 | 127.30.119.65:50109 | HEALTHY 803373f5e35f4a61b5a983ea8b66edca | 127.30.119.68:51058 | UNAVAILABLE Error from 127.30.119.68:51058: Timed out: could not get status from server: Client connection negotiation failed: client connection to 127.30.119.68:51058 (UNAVAILABLE) Flag | Value | Tags | Tablet Server ------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+------------------------- ipki_server_key_size | 1024 | experimental | all 6 server(s) checked local_ip_for_outbound_sockets | 127.30.119.65 | experimental | 127.30.119.65:50109 local_ip_for_outbound_sockets | 127.30.119.66 | experimental | 127.30.119.66:46821 local_ip_for_outbound_sockets | 127.30.119.67 | experimental | 127.30.119.67:58765 local_ip_for_outbound_sockets | 127.30.119.68 | experimental | 127.30.119.68:51058 local_ip_for_outbound_sockets | 127.30.119.69 | experimental | 127.30.119.69:41187 local_ip_for_outbound_sockets | 127.30.119.70 | experimental | 127.30.119.70:44157 never_fsync | true | unsafe,advanced | all 6 server(s) checked raft_prepare_replacement_before_eviction | true | experimental,advanced | all 6 server(s) checked rpc_server_allow_ephemeral_ports | true | unsafe | all 6 server(s) checked server_dump_info_format | pb | hidden | all 6 server(s) checked server_dump_info_path | /data/somelongdirectorytoavoidrpathissues/src/kudutest/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1539775637668737-31197/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.30.119.65:50109 server_dump_info_path | /data/somelongdirectorytoavoidrpathissues/src/kudutest/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1539775637668737-31197/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.30.119.66:46821 server_dump_info_path | /data/somelongdirectorytoavoidrpathissues/src/kudutest/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1539775637668737-31197/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.30.119.67:58765 server_dump_info_path | /data/somelongdirectorytoavoidrpathissues/src/kudutest/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1539775637668737-31197/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.30.119.68:51058 server_dump_info_path | /data/somelongdirectorytoavoidrpathissues/src/kudutest/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1539775637668737-31197/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.30.119.69:41187 server_dump_info_path | /data/somelongdirectorytoavoidrpathissues/src/kudutest/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1539775637668737-31197/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.30.119.70:44157 tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | all 6 server(s) checked Version Summary Version | Servers ----------------+-------------------------------------------------------------------------------------------------------------- 1.9.0-SNAPSHOT | master@127.30.119.126:56502, tserver@127.30.119.67:58765, tserver@127.30.119.70:44157, and 3 other server(s) Tablet 53b982eb47024b95984ad8754627357a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING 88847ecc753c428b8c75790b08f9e33f (127.30.119.69:41187): RUNNING 167e75d199c246a0839f72a87f6b3416 (127.30.119.67:58765): RUNNING [LEADER] 803373f5e35f4a61b5a983ea8b66edca (127.30.119.68:51058): TS unavailable All reported replicas are: A = 88847ecc753c428b8c75790b08f9e33f B = 167e75d199c246a0839f72a87f6b3416 C = 803373f5e35f4a61b5a983ea8b66edca The consensus matrix is: Config source | Replicas | Current term | Config index | Committed? ---------------+------------------------+--------------+--------------+------------ master | A B* C | | | Yes A | A B* C | 1 | 41 | Yes B | A B* C | 1 | 41 | Yes C | [config not available] | | | Tablet c9a30cf14a314d718ce60f43b1fa34f9 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING 167e75d199c246a0839f72a87f6b3416 (127.30.119.67:58765): RUNNING 3ab543a6ac3449a8bdd5c44a04ea5ee6 (127.30.119.70:44157): RUNNING [LEADER] 803373f5e35f4a61b5a983ea8b66edca (127.30.119.68:51058): TS unavailable All reported replicas are: A = 167e75d199c246a0839f72a87f6b3416 B = 3ab543a6ac3449a8bdd5c44a04ea5ee6 C = 803373f5e35f4a61b5a983ea8b66edca The consensus matrix is: Config source | Replicas | Current term | Config index | Committed? ---------------+------------------------+--------------+--------------+------------ master | A B* C | | | Yes A | A B* C | 1 | 43 | Yes B | A B* C | 1 | 43 | Yes C | [config not available] | | | Tablet f430bbb1fef44e65b387432c037df4dd of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING 88847ecc753c428b8c75790b08f9e33f (127.30.119.69:41187): RUNNING [LEADER] 167e75d199c246a0839f72a87f6b3416 (127.30.119.67:58765): RUNNING 803373f5e35f4a61b5a983ea8b66edca (127.30.119.68:51058): TS unavailable All reported replicas are: A = 88847ecc753c428b8c75790b08f9e33f B = 167e75d199c246a0839f72a87f6b3416 C = 803373f5e35f4a61b5a983ea8b66edca The consensus matrix is: Config source | Replicas | Current term | Config index | Committed? ---------------+------------------------+--------------+--------------+------------ master | A* B C | | | Yes A | A* B C | 1 | 45 | Yes B | A* B C | 1 | 45 | Yes C | [config not available] | | | Tablet 2932a59244e646d6a27beb9786b3e9b7 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING 88847ecc753c428b8c75790b08f9e33f (127.30.119.69:41187): RUNNING [LEADER] 167e75d199c246a0839f72a87f6b3416 (127.30.119.67:58765): RUNNING 803373f5e35f4a61b5a983ea8b66edca (127.30.119.68:51058): TS unavailable All reported replicas are: A = 88847ecc753c428b8c75790b08f9e33f B = 167e75d199c246a0839f72a87f6b3416 C = 803373f5e35f4a61b5a983ea8b66edca The consensus matrix is: Config source | Replicas | Current term | Config index | Committed? ---------------+------------------------+--------------+--------------+------------ master | A* B C | | | Yes A | A* B C | 1 | 45 | Yes B | A* B C | 1 | 45 | Yes C | [config not available] | | | Tablet 6d4d96968edc4b73a3b0de435c985e29 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING 3ab543a6ac3449a8bdd5c44a04ea5ee6 (127.30.119.70:44157): RUNNING [LEADER] 167e75d199c246a0839f72a87f6b3416 (127.30.119.67:58765): RUNNING 803373f5e35f4a61b5a983ea8b66edca (127.30.119.68:51058): TS unavailable All reported replicas are: A = 3ab543a6ac3449a8bdd5c44a04ea5ee6 B = 167e75d199c246a0839f72a87f6b3416 C = 803373f5e35f4a61b5a983ea8b66edca The consensus matrix is: Config source | Replicas | Current term | Config index | Committed? ---------------+------------------------+--------------+--------------+------------ master | A* B C | | | Yes A | A* B C | 2 | 46 | Yes B | A* B C | 2 | 46 | Yes C | [config not available] | | | Tablet 4cfb1f5eca244d988010eccdf19b6c10 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING 167e75d199c246a0839f72a87f6b3416 (127.30.119.67:58765): RUNNING 490555426e0148be8a38ccf90bedb317 (127.30.119.66:46821): RUNNING [LEADER] 803373f5e35f4a61b5a983ea8b66edca (127.30.119.68:51058): TS unavailable All reported replicas are: A = 167e75d199c246a0839f72a87f6b3416 B = 490555426e0148be8a38ccf90bedb317 C = 803373f5e35f4a61b5a983ea8b66edca The consensus matrix is: Config source | Replicas | Current term | Config index | Committed? ---------------+------------------------+--------------+--------------+------------ master | A B* C | | | Yes A | A B* C | 1 | 45 | Yes B | A B* C | 1 | 45 | Yes C | [config not available] | | | Tablet 31fde0e543f84efeb62cf97bdc602900 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING 3ab543a6ac3449a8bdd5c44a04ea5ee6 (127.30.119.70:44157): RUNNING 490555426e0148be8a38ccf90bedb317 (127.30.119.66:46821): RUNNING [LEADER] 803373f5e35f4a61b5a983ea8b66edca (127.30.119.68:51058): TS unavailable All reported replicas are: A = 3ab543a6ac3449a8bdd5c44a04ea5ee6 B = 490555426e0148be8a38ccf90bedb317 C = 803373f5e35f4a61b5a983ea8b66edca The consensus matrix is: Config source | Replicas | Current term | Config index | Committed? ---------------+------------------------+--------------+--------------+------------ master | A B* C | | | Yes A | A B* C | 2 | 49 | Yes B | A B* C | 2 | 49 | Yes C | [config not available] | | | Tablet ca302959d1d94e20af56964a27aba4a4 of table 'TestTable' is under-replicated: 1 replica(s) not RUNNING 3ab543a6ac3449a8bdd5c44a04ea5ee6 (127.30.119.70:44157): RUNNING [LEADER] 88847ecc753c428b8c75790b08f9e33f (127.30.119.69:41187): RUNNING 803373f5e35f4a61b5a983ea8b66edca (127.30.119.68:51058): TS unavailable All reported replicas are: A = 3ab543a6ac3449a8bdd5c44a04ea5ee6 B = 88847ecc753c428b8c75790b08f9e33f C = 803373f5e35f4a61b5a983ea8b66edca The consensus matrix is: Config source | Replicas | Current term | Config index | Committed? ---------------+------------------------+--------------+--------------+------------ master | A* B C | | | Yes A | A* B C | 2 | 8 | Yes B | A* B C | 2 | 8 | Yes C | [config not available] | | | Summary by table Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable ----------------------+----+------------------+---------------+---------+------------+------------------+------------- RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 20 | 13 | 0 | 7 | 0 TestTable | 3 | UNDER_REPLICATED | 1 | 0 | 0 | 1 | 0 | Total Count ----------------+------------- Masters | 1 Tablet Servers | 6 Tables | 2 Tablets | 21 Replicas | 63 ================== Warnings: ================== Some masters have unsafe, experimental, or hidden flags set Some tablet servers have unsafe, experimental, or hidden flags set ================== Errors: ================== Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors Corruption: table consistency check error: 2 out of 2 table(s) are not healthy FAILED Master Summary UUID | Address | Status ----------------------------------+----------------------+--------- 0461054002fe43ac852199cb3befad20 | 127.30.119.126:56502 | HEALTHY Flag | Value | Tags | Master ------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+------------------------- ipki_ca_key_size | 1024 | experimental | all 1 server(s) checked ipki_server_key_size | 1024 | experimental | all 1 server(s) checked never_fsync | true | unsafe,advanced | all 1 server(s) checked raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked rpc_reuseport | true | experimental | all 1 server(s) checked rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked server_dump_info_format | pb | hidden | all 1 server(s) checked server_dump_info_path | /data/somelongdirectorytoavoidrpathissues/src/kudutest/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1539775637668737-31197/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked Tablet Server Summary UUID | Address | Status ----------------------------------+---------------------+--------- 167e75d199c246a0839f72a87f6b3416 | 127.30.119.67:58765 | HEALTHY 3ab543a6ac3449a8bdd5c44a04ea5ee6 | 127.30.119.70:44157 | HEALTHY 490555426e0148be8a38ccf90bedb317 | 127.30.119.66:46821 | HEALTHY 803373f5e35f4a61b5a983ea8b66edca | 127.30.119.68:51058 | HEALTHY 88847ecc753c428b8c75790b08f9e33f | 127.30.119.69:41187 | HEALTHY ca720a15d16d4077aa7ae6218d6fbce5 | 127.30.119.65:50109 | HEALTHY Flag | Value | Tags | Tablet Server ------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+------------------------- ipki_server_key_size | 1024 | experimental | all 6 server(s) checked local_ip_for_outbound_sockets | 127.30.119.65 | experimental | 127.30.119.65:50109 local_ip_for_outbound_sockets | 127.30.119.66 | experimental | 127.30.119.66:46821 local_ip_for_outbound_sockets | 127.30.119.67 | experimental | 127.30.119.67:58765 local_ip_for_outbound_sockets | 127.30.119.68 | experimental | 127.30.119.68:51058 local_ip_for_outbound_sockets | 127.30.119.69 | experimental | 127.30.119.69:41187 local_ip_for_outbound_sockets | 127.30.119.70 | experimental | 127.30.119.70:44157 never_fsync | true | unsafe,advanced | all 6 server(s) checked raft_prepare_replacement_before_eviction | true | experimental,advanced | all 6 server(s) checked rpc_server_allow_ephemeral_ports | true | unsafe | all 6 server(s) checked server_dump_info_format | pb | hidden | all 6 server(s) checked server_dump_info_path | /data/somelongdirectorytoavoidrpathissues/src/kudutest/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1539775637668737-31197/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.30.119.65:50109 server_dump_info_path | /data/somelongdirectorytoavoidrpathissues/src/kudutest/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1539775637668737-31197/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.30.119.66:46821 server_dump_info_path | /data/somelongdirectorytoavoidrpathissues/src/kudutest/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1539775637668737-31197/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.30.119.67:58765 server_dump_info_path | /data/somelongdirectorytoavoidrpathissues/src/kudutest/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1539775637668737-31197/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.30.119.68:51058 server_dump_info_path | /data/somelongdirectorytoavoidrpathissues/src/kudutest/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1539775637668737-31197/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.30.119.69:41187 server_dump_info_path | /data/somelongdirectorytoavoidrpathissues/src/kudutest/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1539775637668737-31197/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.30.119.70:44157 tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | all 6 server(s) checked Version Summary Version | Servers ----------------+------------------------- 1.9.0-SNAPSHOT | all 7 server(s) checked Summary by table Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable ----------------------+----+---------+---------------+---------+------------+------------------+------------- RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0 TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0 | Total Count ----------------+------------- Masters | 1 Tablet Servers | 6 Tables | 2 Tablets | 21 Replicas | 63 ================== Warnings: ================== Some masters have unsafe, experimental, or hidden flags set Some tablet servers have unsafe, experimental, or hidden flags set OK I1017 04:27:45.623636 31197 raft_consensus_stress-itest.cc:268] completed iteration 0 E1017 04:27:51.004619 800 consensus_queue.cc:699] T 2932a59244e646d6a27beb9786b3e9b7 P 88847ecc753c428b8c75790b08f9e33f [LEADER]: Error trying to read ahead of the log while preparing peer request: Incomplete: Op with index 55 is ahead of the local log (next sequential op: 53). Destination peer: Peer: permanent_uuid: "ca720a15d16d4077aa7ae6218d6fbce5" member_type: VOTER last_known_addr { host: "127.30.119.65" port: 50109 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.51, Next index: 56, Last known committed idx: 55, Time since last communication: 0.043s E1017 04:27:51.040204 799 consensus_queue.cc:699] T f430bbb1fef44e65b387432c037df4dd P 88847ecc753c428b8c75790b08f9e33f [LEADER]: Error trying to read ahead of the log while preparing peer request: Incomplete: Op with index 55 is ahead of the local log (next sequential op: 53). Destination peer: Peer: permanent_uuid: "ca720a15d16d4077aa7ae6218d6fbce5" member_type: VOTER last_known_addr { host: "127.30.119.65" port: 50109 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.51, Next index: 56, Last known committed idx: 55, Time since last communication: 0.059s E1017 04:27:51.045595 800 consensus_queue.cc:699] T f430bbb1fef44e65b387432c037df4dd P 88847ecc753c428b8c75790b08f9e33f [NON_LEADER]: Error trying to read ahead of the log while preparing peer request: Incomplete: Op with index 55 is ahead of the local log (next sequential op: 53). Destination peer: Peer: permanent_uuid: "ca720a15d16d4077aa7ae6218d6fbce5" member_type: VOTER last_known_addr { host: "127.30.119.65" port: 50109 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.51, Next index: 56, Last known committed idx: 55, Time since last communication: 0.063s E1017 04:27:57.444617 1038 consensus_queue.cc:699] T 9e92fd7d71aa4588aded2ecdd67095cb P 490555426e0148be8a38ccf90bedb317 [LEADER]: Error trying to read ahead of the log while preparing peer request: Incomplete: Op with index 57 is ahead of the local log (next sequential op: 54). Destination peer: Peer: permanent_uuid: "167e75d199c246a0839f72a87f6b3416" member_type: VOTER last_known_addr { host: "127.30.119.67" port: 58765 }, Status: INVALID_TERM, Last received: 2.57, Next index: 58, Last known committed idx: 57, Time since last communication: 0.023s E1017 04:28:03.582799 2159 consensus_queue.cc:699] T bed15e9cdc6d44f69eedb4378cd2ea0f P 167e75d199c246a0839f72a87f6b3416 [LEADER]: Error trying to read ahead of the log while preparing peer request: Incomplete: Op with index 66 is ahead of the local log (next sequential op: 64). Destination peer: Peer: permanent_uuid: "88847ecc753c428b8c75790b08f9e33f" member_type: VOTER last_known_addr { host: "127.30.119.69" port: 41187 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.62, Next index: 67, Last known committed idx: 66, Time since last communication: 0.056s W1017 04:28:06.792627 2260 negotiation.cc:313] Failed RPC negotiation. Trace: 1017 04:28:03.792429 (+ 0us) reactor.cc:583] Submitting negotiation task for client connection to 127.30.119.66:46821 1017 04:28:03.797093 (+ 4664us) negotiation.cc:98] Waiting for socket to connect 1017 04:28:03.797122 (+ 29us) client_negotiation.cc:160] Beginning negotiation 1017 04:28:03.797315 (+ 193us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 1017 04:28:06.792404 (+2995089us) negotiation.cc:304] Negotiation complete: Timed out: Client connection negotiation failed: client connection to 127.30.119.66:46821 Metrics: {"client-negotiator.queue_time_us":4379,"thread_start_us":3656,"threads_started":1} Master Summary UUID | Address | Status ----------------------------------+----------------------+--------- 0461054002fe43ac852199cb3befad20 | 127.30.119.126:56502 | HEALTHY Flag | Value | Tags | Master ------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+------------------------- ipki_ca_key_size | 1024 | experimental | all 1 server(s) checked ipki_server_key_size | 1024 | experimental | all 1 server(s) checked never_fsync | true | unsafe,advanced | all 1 server(s) checked raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked rpc_reuseport | true | experimental | all 1 server(s) checked rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked server_dump_info_format | pb | hidden | all 1 server(s) checked server_dump_info_path | /data/somelongdirectorytoavoidrpathissues/src/kudutest/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1539775637668737-31197/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked Tablet Server Summary UUID | Address | Status ----------------------------------+---------------------+------------- 167e75d199c246a0839f72a87f6b3416 | 127.30.119.67:58765 | HEALTHY 3ab543a6ac3449a8bdd5c44a04ea5ee6 | 127.30.119.70:44157 | HEALTHY 803373f5e35f4a61b5a983ea8b66edca | 127.30.119.68:51058 | HEALTHY 88847ecc753c428b8c75790b08f9e33f | 127.30.119.69:41187 | HEALTHY ca720a15d16d4077aa7ae6218d6fbce5 | 127.30.119.65:50109 | HEALTHY 490555426e0148be8a38ccf90bedb317 | 127.30.119.66:46821 | UNAVAILABLE Error from 127.30.119.66:46821: Timed out: could not get status from server: Client connection negotiation failed: client connection to 127.30.119.66:46821 (UNAVAILABLE) Flag | Value | Tags | Tablet Server ------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+------------------------- ipki_server_key_size | 1024 | experimental | all 6 server(s) checked local_ip_for_outbound_sockets | 127.30.119.65 | experimental | 127.30.119.65:50109 local_ip_for_outbound_sockets | 127.30.119.66 | experimental | 127.30.119.66:46821 local_ip_for_outbound_sockets | 127.30.119.67 | experimental | 127.30.119.67:58765 local_ip_for_outbound_sockets | 127.30.119.68 | experimental | 127.30.119.68:51058 local_ip_for_outbound_sockets | 127.30.119.69 | experimental | 127.30.119.69:41187 local_ip_for_outbound_sockets | 127.30.119.70 | experimental | 127.30.119.70:44157 never_fsync | true | unsafe,advanced | all 6 server(s) checked raft_prepare_replacement_before_eviction | true | experimental,advanced | all 6 server(s) checked rpc_server_allow_ephemeral_ports | true | unsafe | all 6 server(s) checked server_dump_info_format | pb | hidden | all 6 server(s) checked server_dump_info_path | /data/somelongdirectorytoavoidrpathissues/src/kudutest/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1539775637668737-31197/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.30.119.65:50109 server_dump_info_path | /data/somelongdirectorytoavoidrpathissues/src/kudutest/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1539775637668737-31197/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.30.119.66:46821 server_dump_info_path | /data/somelongdirectorytoavoidrpathissues/src/kudutest/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1539775637668737-31197/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.30.119.67:58765 server_dump_info_path | /data/somelongdirectorytoavoidrpathissues/src/kudutest/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1539775637668737-31197/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.30.119.68:51058 server_dump_info_path | /data/somelongdirectorytoavoidrpathissues/src/kudutest/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1539775637668737-31197/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.30.119.69:41187 server_dump_info_path | /data/somelongdirectorytoavoidrpathissues/src/kudutest/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1539775637668737-31197/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.30.119.70:44157 tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | all 6 server(s) checked Version Summary Version | Servers ----------------+-------------------------------------------------------------------------------------------------------------- 1.9.0-SNAPSHOT | master@127.30.119.126:56502, tserver@127.30.119.67:58765, tserver@127.30.119.70:44157, and 3 other server(s) Tablet 53b982eb47024b95984ad8754627357a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING ca720a15d16d4077aa7ae6218d6fbce5 (127.30.119.65:50109): RUNNING 803373f5e35f4a61b5a983ea8b66edca (127.30.119.68:51058): RUNNING [LEADER] 490555426e0148be8a38ccf90bedb317 (127.30.119.66:46821): TS unavailable All reported replicas are: A = ca720a15d16d4077aa7ae6218d6fbce5 B = 803373f5e35f4a61b5a983ea8b66edca C = 490555426e0148be8a38ccf90bedb317 The consensus matrix is: Config source | Replicas | Current term | Config index | Committed? ---------------+------------------------+--------------+--------------+------------ master | A B* C | | | Yes A | A B* C | 2 | 56 | Yes B | A B* C | 2 | 56 | Yes C | [config not available] | | | Tablet c9a30cf14a314d718ce60f43b1fa34f9 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING 3ab543a6ac3449a8bdd5c44a04ea5ee6 (127.30.119.70:44157): RUNNING [LEADER] ca720a15d16d4077aa7ae6218d6fbce5 (127.30.119.65:50109): RUNNING 490555426e0148be8a38ccf90bedb317 (127.30.119.66:46821): TS unavailable All reported replicas are: A = 3ab543a6ac3449a8bdd5c44a04ea5ee6 B = ca720a15d16d4077aa7ae6218d6fbce5 C = 490555426e0148be8a38ccf90bedb317 The consensus matrix is: Config source | Replicas | Current term | Config index | Committed? ---------------+------------------------+--------------+--------------+------------ master | A* B C | | | Yes A | A* B C | 3 | 57 | Yes B | A* B C | 3 | 57 | Yes C | [config not available] | | | Tablet 26fe0fdb98f64980acd487056a22149f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING ca720a15d16d4077aa7ae6218d6fbce5 (127.30.119.65:50109): RUNNING [LEADER] 88847ecc753c428b8c75790b08f9e33f (127.30.119.69:41187): RUNNING 490555426e0148be8a38ccf90bedb317 (127.30.119.66:46821): TS unavailable All reported replicas are: A = ca720a15d16d4077aa7ae6218d6fbce5 B = 88847ecc753c428b8c75790b08f9e33f C = 490555426e0148be8a38ccf90bedb317 The consensus matrix is: Config source | Replicas | Current term | Config index | Committed? ---------------+------------------------+--------------+--------------+------------ master | A* B C | | | Yes A | A* B C | 2 | 58 | Yes B | A* B C | 2 | 58 | Yes C | [config not available] | | | Tablet 9e92fd7d71aa4588aded2ecdd67095cb of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING ca720a15d16d4077aa7ae6218d6fbce5 (127.30.119.65:50109): RUNNING [LEADER] 88847ecc753c428b8c75790b08f9e33f (127.30.119.69:41187): RUNNING 490555426e0148be8a38ccf90bedb317 (127.30.119.66:46821): TS unavailable All reported replicas are: A = ca720a15d16d4077aa7ae6218d6fbce5 B = 88847ecc753c428b8c75790b08f9e33f C = 490555426e0148be8a38ccf90bedb317 The consensus matrix is: Config source | Replicas | Current term | Config index | Committed? ---------------+------------------------+--------------+--------------+------------ master | A* B C | | | Yes A | A* B C | 2 | 61 | Yes B | A* B C | 2 | 61 | Yes C | [config not available] | | | Tablet ce185e91fbc5412fa8bfb9c86df7711b of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING 3ab543a6ac3449a8bdd5c44a04ea5ee6 (127.30.119.70:44157): RUNNING [LEADER] 88847ecc753c428b8c75790b08f9e33f (127.30.119.69:41187): RUNNING 490555426e0148be8a38ccf90bedb317 (127.30.119.66:46821): TS unavailable All reported replicas are: A = 3ab543a6ac3449a8bdd5c44a04ea5ee6 B = 88847ecc753c428b8c75790b08f9e33f C = 490555426e0148be8a38ccf90bedb317 The consensus matrix is: Config source | Replicas | Current term | Config index | Committed? ---------------+------------------------+--------------+--------------+------------ master | A* B C | | | Yes A | A* B C | 4 | 60 | Yes B | A* B C | 4 | 60 | Yes C | [config not available] | | | Tablet f430bbb1fef44e65b387432c037df4dd of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING ca720a15d16d4077aa7ae6218d6fbce5 (127.30.119.65:50109): RUNNING [LEADER] 803373f5e35f4a61b5a983ea8b66edca (127.30.119.68:51058): RUNNING 490555426e0148be8a38ccf90bedb317 (127.30.119.66:46821): TS unavailable All reported replicas are: A = ca720a15d16d4077aa7ae6218d6fbce5 B = 803373f5e35f4a61b5a983ea8b66edca C = 490555426e0148be8a38ccf90bedb317 The consensus matrix is: Config source | Replicas | Current term | Config index | Committed? ---------------+------------------------+--------------+--------------+------------ master | A* B C | | | Yes A | A* B C | 2 | 60 | Yes B | A* B C | 2 | 60 | Yes C | [config not available] | | | Tablet 9dd66394bfb145828597c3d9f214af89 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING 803373f5e35f4a61b5a983ea8b66edca (127.30.119.68:51058): RUNNING [LEADER] ca720a15d16d4077aa7ae6218d6fbce5 (127.30.119.65:50109): RUNNING 490555426e0148be8a38ccf90bedb317 (127.30.119.66:46821): TS unavailable All reported replicas are: A = 803373f5e35f4a61b5a983ea8b66edca B = ca720a15d16d4077aa7ae6218d6fbce5 C = 490555426e0148be8a38ccf90bedb317 The consensus matrix is: Config source | Replicas | Current term | Config index | Committed? ---------------+------------------------+--------------+--------------+------------ master | A* B C | | | Yes A | A* B C | 2 | 54 | Yes B | A* B C | 2 | 54 | Yes C | [config not available] | | | Tablet 76428cfea0ba491fb3d4acf4b994d6a1 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING 3ab543a6ac3449a8bdd5c44a04ea5ee6 (127.30.119.70:44157): RUNNING 803373f5e35f4a61b5a983ea8b66edca (127.30.119.68:51058): RUNNING [LEADER] 490555426e0148be8a38ccf90bedb317 (127.30.119.66:46821): TS unavailable All reported replicas are: A = 3ab543a6ac3449a8bdd5c44a04ea5ee6 B = 803373f5e35f4a61b5a983ea8b66edca C = 490555426e0148be8a38ccf90bedb317 The consensus matrix is: Config source | Replicas | Current term | Config index | Committed? ---------------+------------------------+--------------+--------------+------------ master | A B* C | | | Yes A | A B* C | 2 | 53 | Yes B | A B* C | 2 | 53 | Yes C | [config not available] | | | Summary by table Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable ----------------------+----+------------------+---------------+---------+------------+------------------+------------- TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0 RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 20 | 12 | 0 | 8 | 0 | Total Count ----------------+------------- Masters | 1 Tablet Servers | 6 Tables | 2 Tablets | 21 Replicas | 63 ================== Warnings: ================== Some masters have unsafe, experimental, or hidden flags set Some tablet servers have unsafe, experimental, or hidden flags set ================== Errors: ================== Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors Corruption: table consistency check error: 1 out of 2 table(s) are not healthy FAILED Master Summary UUID | Address | Status ----------------------------------+----------------------+--------- 0461054002fe43ac852199cb3befad20 | 127.30.119.126:56502 | HEALTHY Flag | Value | Tags | Master ------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+------------------------- ipki_ca_key_size | 1024 | experimental | all 1 server(s) checked ipki_server_key_size | 1024 | experimental | all 1 server(s) checked never_fsync | true | unsafe,advanced | all 1 server(s) checked raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked rpc_reuseport | true | experimental | all 1 server(s) checked rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked server_dump_info_format | pb | hidden | all 1 server(s) checked server_dump_info_path | /data/somelongdirectorytoavoidrpathissues/src/kudutest/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1539775637668737-31197/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked Tablet Server Summary UUID | Address | Status ----------------------------------+---------------------+--------- 167e75d199c246a0839f72a87f6b3416 | 127.30.119.67:58765 | HEALTHY 3ab543a6ac3449a8bdd5c44a04ea5ee6 | 127.30.119.70:44157 | HEALTHY 490555426e0148be8a38ccf90bedb317 | 127.30.119.66:46821 | HEALTHY 803373f5e35f4a61b5a983ea8b66edca | 127.30.119.68:51058 | HEALTHY 88847ecc753c428b8c75790b08f9e33f | 127.30.119.69:41187 | HEALTHY ca720a15d16d4077aa7ae6218d6fbce5 | 127.30.119.65:50109 | HEALTHY Flag | Value | Tags | Tablet Server ------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+------------------------- ipki_server_key_size | 1024 | experimental | all 6 server(s) checked local_ip_for_outbound_sockets | 127.30.119.65 | experimental | 127.30.119.65:50109 local_ip_for_outbound_sockets | 127.30.119.66 | experimental | 127.30.119.66:46821 local_ip_for_outbound_sockets | 127.30.119.67 | experimental | 127.30.119.67:58765 local_ip_for_outbound_sockets | 127.30.119.68 | experimental | 127.30.119.68:51058 local_ip_for_outbound_sockets | 127.30.119.69 | experimental | 127.30.119.69:41187 local_ip_for_outbound_sockets | 127.30.119.70 | experimental | 127.30.119.70:44157 never_fsync | true | unsafe,advanced | all 6 server(s) checked raft_prepare_replacement_before_eviction | true | experimental,advanced | all 6 server(s) checked rpc_server_allow_ephemeral_ports | true | unsafe | all 6 server(s) checked server_dump_info_format | pb | hidden | all 6 server(s) checked server_dump_info_path | /data/somelongdirectorytoavoidrpathissues/src/kudutest/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1539775637668737-31197/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.30.119.65:50109 server_dump_info_path | /data/somelongdirectorytoavoidrpathissues/src/kudutest/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1539775637668737-31197/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.30.119.66:46821 server_dump_info_path | /data/somelongdirectorytoavoidrpathissues/src/kudutest/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1539775637668737-31197/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.30.119.67:58765 server_dump_info_path | /data/somelongdirectorytoavoidrpathissues/src/kudutest/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1539775637668737-31197/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.30.119.68:51058 server_dump_info_path | /data/somelongdirectorytoavoidrpathissues/src/kudutest/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1539775637668737-31197/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.30.119.69:41187 server_dump_info_path | /data/somelongdirectorytoavoidrpathissues/src/kudutest/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1539775637668737-31197/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.30.119.70:44157 tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | all 6 server(s) checked Version Summary Version | Servers ----------------+------------------------- 1.9.0-SNAPSHOT | all 7 server(s) checked Summary by table Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable ----------------------+----+---------+---------------+---------+------------+------------------+------------- RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0 TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0 | Total Count ----------------+------------- Masters | 1 Tablet Servers | 6 Tables | 2 Tablets | 21 Replicas | 63 ================== Warnings: ================== Some masters have unsafe, experimental, or hidden flags set Some tablet servers have unsafe, experimental, or hidden flags set OK I1017 04:28:11.645534 31197 raft_consensus_stress-itest.cc:268] completed iteration 1 E1017 04:28:29.542846 17536 consensus_queue.cc:699] T ce185e91fbc5412fa8bfb9c86df7711b P 88847ecc753c428b8c75790b08f9e33f [LEADER]: Error trying to read ahead of the log while preparing peer request: Incomplete: Op with index 75 is ahead of the local log (next sequential op: 73). Destination peer: Peer: permanent_uuid: "490555426e0148be8a38ccf90bedb317" member_type: VOTER last_known_addr { host: "127.30.119.66" port: 46821 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.71, Next index: 76, Last known committed idx: 75, Time since last communication: 0.065s E1017 04:28:29.547380 17539 consensus_queue.cc:699] T ce185e91fbc5412fa8bfb9c86df7711b P 88847ecc753c428b8c75790b08f9e33f [LEADER]: Error trying to read ahead of the log while preparing peer request: Incomplete: Op with index 75 is ahead of the local log (next sequential op: 73). Destination peer: Peer: permanent_uuid: "167e75d199c246a0839f72a87f6b3416" member_type: VOTER last_known_addr { host: "127.30.119.67" port: 58765 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.75, Next index: 76, Last known committed idx: 75, Time since last communication: 0.028s W1017 04:28:32.744110 17618 negotiation.cc:313] Failed RPC negotiation. Trace: 1017 04:28:29.743642 (+ 0us) reactor.cc:583] Submitting negotiation task for client connection to 127.30.119.67:58765 1017 04:28:29.743973 (+ 331us) negotiation.cc:98] Waiting for socket to connect 1017 04:28:29.743989 (+ 16us) client_negotiation.cc:160] Beginning negotiation 1017 04:28:29.744103 (+ 114us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 1017 04:28:32.743403 (+2999300us) negotiation.cc:304] Negotiation complete: Timed out: Client connection negotiation failed: client connection to 127.30.119.67:58765 Metrics: {"client-negotiator.queue_time_us":174} W1017 04:28:35.745781 17618 negotiation.cc:313] Failed RPC negotiation. Trace: 1017 04:28:32.744722 (+ 0us) reactor.cc:583] Submitting negotiation task for client connection to 127.30.119.67:58765 1017 04:28:32.744915 (+ 193us) negotiation.cc:98] Waiting for socket to connect 1017 04:28:32.744932 (+ 17us) client_negotiation.cc:160] Beginning negotiation 1017 04:28:32.745042 (+ 110us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 1017 04:28:35.728094 (+2983052us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 1017 04:28:35.728102 (+ 8us) client_negotiation.cc:348] Received NEGOTIATE response from server 1017 04:28:35.728169 (+ 67us) client_negotiation.cc:175] Negotiated authn=SASL 1017 04:28:35.728312 (+ 143us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 1017 04:28:35.728409 (+ 97us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 1017 04:28:35.728425 (+ 16us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 1017 04:28:35.744468 (+ 16043us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 1017 04:28:35.744475 (+ 7us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 1017 04:28:35.745171 (+ 696us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 1017 04:28:35.745215 (+ 44us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 1017 04:28:35.745486 (+ 271us) negotiation.cc:304] Negotiation complete: Timed out: Client connection negotiation failed: client connection to 127.30.119.67:58765: BlockingWrite timed out Metrics: {"client-negotiator.queue_time_us":53} Master Summary UUID | Address | Status ----------------------------------+----------------------+--------- 0461054002fe43ac852199cb3befad20 | 127.30.119.126:56502 | HEALTHY Flag | Value | Tags | Master ------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+------------------------- ipki_ca_key_size | 1024 | experimental | all 1 server(s) checked ipki_server_key_size | 1024 | experimental | all 1 server(s) checked never_fsync | true | unsafe,advanced | all 1 server(s) checked raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked rpc_reuseport | true | experimental | all 1 server(s) checked rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked server_dump_info_format | pb | hidden | all 1 server(s) checked server_dump_info_path | /data/somelongdirectorytoavoidrpathissues/src/kudutest/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1539775637668737-31197/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked Tablet Server Summary UUID | Address | Status ----------------------------------+---------------------+------------- 3ab543a6ac3449a8bdd5c44a04ea5ee6 | 127.30.119.70:44157 | HEALTHY 490555426e0148be8a38ccf90bedb317 | 127.30.119.66:46821 | HEALTHY 803373f5e35f4a61b5a983ea8b66edca | 127.30.119.68:51058 | HEALTHY 88847ecc753c428b8c75790b08f9e33f | 127.30.119.69:41187 | HEALTHY ca720a15d16d4077aa7ae6218d6fbce5 | 127.30.119.65:50109 | HEALTHY 167e75d199c246a0839f72a87f6b3416 | 127.30.119.67:58765 | UNAVAILABLE Error from 127.30.119.67:58765: Timed out: could not get status from server: Client connection negotiation failed: client connection to 127.30.119.67:58765 (UNAVAILABLE) Flag | Value | Tags | Tablet Server ------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------------------------------------------------------------------- ipki_server_key_size | 1024 | experimental | 127.30.119.70:44157, 127.30.119.66:46821, 127.30.119.68:51058, and 2 other server(s) local_ip_for_outbound_sockets | 127.30.119.65 | experimental | 127.30.119.65:50109 local_ip_for_outbound_sockets | 127.30.119.66 | experimental | 127.30.119.66:46821 local_ip_for_outbound_sockets | 127.30.119.68 | experimental | 127.30.119.68:51058 local_ip_for_outbound_sockets | 127.30.119.69 | experimental | 127.30.119.69:41187 local_ip_for_outbound_sockets | 127.30.119.70 | experimental | 127.30.119.70:44157 never_fsync | true | unsafe,advanced | 127.30.119.70:44157, 127.30.119.66:46821, 127.30.119.68:51058, and 2 other server(s) raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.30.119.70:44157, 127.30.119.66:46821, 127.30.119.68:51058, and 2 other server(s) rpc_server_allow_ephemeral_ports | true | unsafe | 127.30.119.70:44157, 127.30.119.66:46821, 127.30.119.68:51058, and 2 other server(s) server_dump_info_format | pb | hidden | 127.30.119.70:44157, 127.30.119.66:46821, 127.30.119.68:51058, and 2 other server(s) server_dump_info_path | /data/somelongdirectorytoavoidrpathissues/src/kudutest/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1539775637668737-31197/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.30.119.65:50109 server_dump_info_path | /data/somelongdirectorytoavoidrpathissues/src/kudutest/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1539775637668737-31197/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.30.119.66:46821 server_dump_info_path | /data/somelongdirectorytoavoidrpathissues/src/kudutest/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1539775637668737-31197/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.30.119.68:51058 server_dump_info_path | /data/somelongdirectorytoavoidrpathissues/src/kudutest/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1539775637668737-31197/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.30.119.69:41187 server_dump_info_path | /data/somelongdirectorytoavoidrpathissues/src/kudutest/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1539775637668737-31197/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.30.119.70:44157 tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.30.119.70:44157, 127.30.119.66:46821, 127.30.119.68:51058, and 2 other server(s) Version Summary Version | Servers ----------------+-------------------------------------------------------------------------------------------------------------- 1.9.0-SNAPSHOT | master@127.30.119.126:56502, tserver@127.30.119.70:44157, tserver@127.30.119.66:46821, and 3 other server(s) Tablet 53b982eb47024b95984ad8754627357a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING ca720a15d16d4077aa7ae6218d6fbce5 (127.30.119.65:50109): RUNNING 167e75d199c246a0839f72a87f6b3416 (127.30.119.67:58765): TS unavailable [LEADER] 803373f5e35f4a61b5a983ea8b66edca (127.30.119.68:51058): RUNNING All reported replicas are: A = ca720a15d16d4077aa7ae6218d6fbce5 B = 167e75d199c246a0839f72a87f6b3416 C = 803373f5e35f4a61b5a983ea8b66edca The consensus matrix is: Config source | Replicas | Current term | Config index | Committed? ---------------+------------------------+--------------+--------------+------------ master | A B* C | | | Yes A | A B* C | 3 | 70 | Yes B | [config not available] | | | C | A B* C | 3 | 70 | Yes Tablet 64d62b98e0284c938ed14bad999a14e3 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING 490555426e0148be8a38ccf90bedb317 (127.30.119.66:46821): RUNNING 167e75d199c246a0839f72a87f6b3416 (127.30.119.67:58765): TS unavailable [LEADER] 3ab543a6ac3449a8bdd5c44a04ea5ee6 (127.30.119.70:44157): RUNNING All reported replicas are: A = 490555426e0148be8a38ccf90bedb317 B = 167e75d199c246a0839f72a87f6b3416 C = 3ab543a6ac3449a8bdd5c44a04ea5ee6 The consensus matrix is: Config source | Replicas | Current term | Config index | Committed? ---------------+------------------------+--------------+--------------+------------ master | A B* C | | | Yes A | A B* C | 4 | 69 | Yes B | [config not available] | | | C | A B* C | 4 | 69 | Yes Tablet c9a30cf14a314d718ce60f43b1fa34f9 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING ca720a15d16d4077aa7ae6218d6fbce5 (127.30.119.65:50109): RUNNING 167e75d199c246a0839f72a87f6b3416 (127.30.119.67:58765): TS unavailable 803373f5e35f4a61b5a983ea8b66edca (127.30.119.68:51058): RUNNING [LEADER] All reported replicas are: A = ca720a15d16d4077aa7ae6218d6fbce5 B = 167e75d199c246a0839f72a87f6b3416 C = 803373f5e35f4a61b5a983ea8b66edca The consensus matrix is: Config source | Replicas | Current term | Config index | Committed? ---------------+------------------------+--------------+--------------+------------ master | A B C* | | | Yes A | A B C* | 5 | 67 | Yes B | [config not available] | | | C | A B C* | 5 | 67 | Yes Tablet 26fe0fdb98f64980acd487056a22149f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING ca720a15d16d4077aa7ae6218d6fbce5 (127.30.119.65:50109): RUNNING [LEADER] 167e75d199c246a0839f72a87f6b3416 (127.30.119.67:58765): TS unavailable 803373f5e35f4a61b5a983ea8b66edca (127.30.119.68:51058): RUNNING All reported replicas are: A = ca720a15d16d4077aa7ae6218d6fbce5 B = 167e75d199c246a0839f72a87f6b3416 C = 803373f5e35f4a61b5a983ea8b66edca The consensus matrix is: Config source | Replicas | Current term | Config index | Committed? ---------------+------------------------+--------------+--------------+------------ master | A* B C | | | Yes A | A* B C | 2 | 68 | Yes B | [config not available] | | | C | A* B C | 2 | 68 | Yes Tablet ce185e91fbc5412fa8bfb9c86df7711b of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING 167e75d199c246a0839f72a87f6b3416 (127.30.119.67:58765): TS unavailable 490555426e0148be8a38ccf90bedb317 (127.30.119.66:46821): RUNNING [LEADER] 3ab543a6ac3449a8bdd5c44a04ea5ee6 (127.30.119.70:44157): RUNNING All reported replicas are: A = 167e75d199c246a0839f72a87f6b3416 B = 490555426e0148be8a38ccf90bedb317 C = 3ab543a6ac3449a8bdd5c44a04ea5ee6 The consensus matrix is: Config source | Replicas | Current term | Config index | Committed? ---------------+------------------------+--------------+--------------+------------ master | A B* C | | | Yes A | [config not available] | | | B | A B* C | 6 | 75 | Yes C | A B* C | 6 | 75 | Yes Tablet 265861d9538a4feaa5de3c1bf3af5a1a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING 167e75d199c246a0839f72a87f6b3416 (127.30.119.67:58765): TS unavailable [LEADER] 803373f5e35f4a61b5a983ea8b66edca (127.30.119.68:51058): RUNNING 3ab543a6ac3449a8bdd5c44a04ea5ee6 (127.30.119.70:44157): RUNNING All reported replicas are: A = 167e75d199c246a0839f72a87f6b3416 B = 803373f5e35f4a61b5a983ea8b66edca C = 3ab543a6ac3449a8bdd5c44a04ea5ee6 The consensus matrix is: Config source | Replicas | Current term | Config index | Committed? ---------------+------------------------+--------------+--------------+------------ master | A* B C | | | Yes A | [config not available] | | | B | A* B C | 4 | 71 | Yes C | A* B C | 4 | 71 | Yes Tablet f430bbb1fef44e65b387432c037df4dd of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING ca720a15d16d4077aa7ae6218d6fbce5 (127.30.119.65:50109): RUNNING [LEADER] 167e75d199c246a0839f72a87f6b3416 (127.30.119.67:58765): TS unavailable 490555426e0148be8a38ccf90bedb317 (127.30.119.66:46821): RUNNING All reported replicas are: A = ca720a15d16d4077aa7ae6218d6fbce5 B = 167e75d199c246a0839f72a87f6b3416 C = 490555426e0148be8a38ccf90bedb317 The consensus matrix is: Config source | Replicas | Current term | Config index | Committed? ---------------+------------------------+--------------+--------------+------------ master | A* B C | | | Yes A | A* B C | 2 | 69 | Yes B | [config not available] | | | C | A* B C | 2 | 69 | Yes Tablet bed15e9cdc6d44f69eedb4378cd2ea0f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING ca720a15d16d4077aa7ae6218d6fbce5 (127.30.119.65:50109): RUNNING [LEADER] 167e75d199c246a0839f72a87f6b3416 (127.30.119.67:58765): TS unavailable 3ab543a6ac3449a8bdd5c44a04ea5ee6 (127.30.119.70:44157): RUNNING All reported replicas are: A = ca720a15d16d4077aa7ae6218d6fbce5 B = 167e75d199c246a0839f72a87f6b3416 C = 3ab543a6ac3449a8bdd5c44a04ea5ee6 The consensus matrix is: Config source | Replicas | Current term | Config index | Committed? ---------------+------------------------+--------------+--------------+------------ master | A* B C | | | Yes A | A* B C | 7 | 78 | Yes B | [config not available] | | | C | A* B C | 7 | 78 | Yes Tablet 9dd66394bfb145828597c3d9f214af89 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING ca720a15d16d4077aa7ae6218d6fbce5 (127.30.119.65:50109): RUNNING [LEADER] 167e75d199c246a0839f72a87f6b3416 (127.30.119.67:58765): TS unavailable 490555426e0148be8a38ccf90bedb317 (127.30.119.66:46821): RUNNING All reported replicas are: A = ca720a15d16d4077aa7ae6218d6fbce5 B = 167e75d199c246a0839f72a87f6b3416 C = 490555426e0148be8a38ccf90bedb317 The consensus matrix is: Config source | Replicas | Current term | Config index | Committed? ---------------+------------------------+--------------+--------------+------------ master | A* B C | | | Yes A | A* B C | 4 | 68 | Yes B | [config not available] | | | C | A* B C | 4 | 68 | Yes Tablet f3233d2d28a74618a2ffcf7c41bc827d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING ca720a15d16d4077aa7ae6218d6fbce5 (127.30.119.65:50109): RUNNING 167e75d199c246a0839f72a87f6b3416 (127.30.119.67:58765): TS unavailable [LEADER] 490555426e0148be8a38ccf90bedb317 (127.30.119.66:46821): RUNNING All reported replicas are: A = ca720a15d16d4077aa7ae6218d6fbce5 B = 167e75d199c246a0839f72a87f6b3416 C = 490555426e0148be8a38ccf90bedb317 The consensus matrix is: Config source | Replicas | Current term | Config index | Committed? ---------------+------------------------+--------------+--------------+------------ master | A B* C | | | Yes A | A B* C | 4 | 64 | Yes B | [config not available] | | | C | A B* C | 4 | 64 | Yes Tablet f6a1e2e405e3449bb0f1fd86a153b0b1 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING 167e75d199c246a0839f72a87f6b3416 (127.30.119.67:58765): TS unavailable 803373f5e35f4a61b5a983ea8b66edca (127.30.119.68:51058): RUNNING [LEADER] 490555426e0148be8a38ccf90bedb317 (127.30.119.66:46821): RUNNING All reported replicas are: A = 167e75d199c246a0839f72a87f6b3416 B = 803373f5e35f4a61b5a983ea8b66edca C = 490555426e0148be8a38ccf90bedb317 The consensus matrix is: Config source | Replicas | Current term | Config index | Committed? ---------------+------------------------+--------------+--------------+------------ master | A B* C | | | Yes A | [config not available] | | | B | A B* C | 6 | 74 | Yes C | A B* C | 6 | 74 | Yes Tablet 8b03926386a44d589f148dbeb73e17ca of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING ca720a15d16d4077aa7ae6218d6fbce5 (127.30.119.65:50109): RUNNING [LEADER] 167e75d199c246a0839f72a87f6b3416 (127.30.119.67:58765): TS unavailable 803373f5e35f4a61b5a983ea8b66edca (127.30.119.68:51058): RUNNING All reported replicas are: A = ca720a15d16d4077aa7ae6218d6fbce5 B = 167e75d199c246a0839f72a87f6b3416 C = 803373f5e35f4a61b5a983ea8b66edca The consensus matrix is: Config source | Replicas | Current term | Config index | Committed? ---------------+------------------------+--------------+--------------+------------ master | A* B C | | | Yes A | A* B C | 7 | 73 | Yes B | [config not available] | | | C | A* B C | 7 | 73 | Yes Tablet 76428cfea0ba491fb3d4acf4b994d6a1 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING 167e75d199c246a0839f72a87f6b3416 (127.30.119.67:58765): TS unavailable [LEADER] 490555426e0148be8a38ccf90bedb317 (127.30.119.66:46821): RUNNING 803373f5e35f4a61b5a983ea8b66edca (127.30.119.68:51058): RUNNING All reported replicas are: A = 167e75d199c246a0839f72a87f6b3416 B = 490555426e0148be8a38ccf90bedb317 C = 803373f5e35f4a61b5a983ea8b66edca The consensus matrix is: Config source | Replicas | Current term | Config index | Committed? ---------------+------------------------+--------------+--------------+------------ master | A* B C | | | Yes A | [config not available] | | | B | A* B C | 4 | 67 | Yes C | A* B C | 4 | 67 | Yes Tablet 4cfb1f5eca244d988010eccdf19b6c10 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING 490555426e0148be8a38ccf90bedb317 (127.30.119.66:46821): RUNNING [LEADER] 167e75d199c246a0839f72a87f6b3416 (127.30.119.67:58765): TS unavailable 3ab543a6ac3449a8bdd5c44a04ea5ee6 (127.30.119.70:44157): RUNNING All reported replicas are: A = 490555426e0148be8a38ccf90bedb317 B = 167e75d199c246a0839f72a87f6b3416 C = 3ab543a6ac3449a8bdd5c44a04ea5ee6 The consensus matrix is: Config source | Replicas | Current term | Config index | Committed? ---------------+------------------------+--------------+--------------+------------ master | A* B C | | | Yes A | A* B C | 7 | 78 | Yes B | [config not available] | | | C | A* B C | 7 | 78 | Yes Tablet ca302959d1d94e20af56964a27aba4a4 of table 'TestTable' is under-replicated: 1 replica(s) not RUNNING ca720a15d16d4077aa7ae6218d6fbce5 (127.30.119.65:50109): RUNNING [LEADER] 490555426e0148be8a38ccf90bedb317 (127.30.119.66:46821): RUNNING 167e75d199c246a0839f72a87f6b3416 (127.30.119.67:58765): TS unavailable All reported replicas are: A = ca720a15d16d4077aa7ae6218d6fbce5 B = 490555426e0148be8a38ccf90bedb317 C = 167e75d199c246a0839f72a87f6b3416 The consensus matrix is: Config source | Replicas | Current term | Config index | Committed? ---------------+------------------------+--------------+--------------+------------ master | A* B C | | | Yes A | A* B C | 3 | 21 | Yes B | A* B C | 3 | 21 | Yes C | [config not available] | | | Summary by table Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable ----------------------+----+------------------+---------------+---------+------------+------------------+------------- RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 20 | 6 | 0 | 14 | 0 TestTable | 3 | UNDER_REPLICATED | 1 | 0 | 0 | 1 | 0 | Total Count ----------------+------------- Masters | 1 Tablet Servers | 6 Tables | 2 Tablets | 21 Replicas | 63 ================== Warnings: ================== Some masters have unsafe, experimental, or hidden flags set unable to get flag information for tablet server 167e75d199c246a0839f72a87f6b3416 (127.30.119.67:58765): Client connection negotiation failed: client connection to 127.30.119.67:58765: BlockingWrite timed out Some tablet servers have unsafe, experimental, or hidden flags set tserver flag check error: 1 of 6 tservers' flags were not available ================== Errors: ================== Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors Corruption: table consistency check error: 2 out of 2 table(s) are not healthy FAILED Master Summary UUID | Address | Status ----------------------------------+----------------------+--------- 0461054002fe43ac852199cb3befad20 | 127.30.119.126:56502 | HEALTHY Flag | Value | Tags | Master ------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+------------------------- ipki_ca_key_size | 1024 | experimental | all 1 server(s) checked ipki_server_key_size | 1024 | experimental | all 1 server(s) checked never_fsync | true | unsafe,advanced | all 1 server(s) checked raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked rpc_reuseport | true | experimental | all 1 server(s) checked rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked server_dump_info_format | pb | hidden | all 1 server(s) checked server_dump_info_path | /data/somelongdirectorytoavoidrpathissues/src/kudutest/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1539775637668737-31197/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked Tablet Server Summary UUID | Address | Status ----------------------------------+---------------------+--------- 167e75d199c246a0839f72a87f6b3416 | 127.30.119.67:58765 | HEALTHY 3ab543a6ac3449a8bdd5c44a04ea5ee6 | 127.30.119.70:44157 | HEALTHY 490555426e0148be8a38ccf90bedb317 | 127.30.119.66:46821 | HEALTHY 803373f5e35f4a61b5a983ea8b66edca | 127.30.119.68:51058 | HEALTHY 88847ecc753c428b8c75790b08f9e33f | 127.30.119.69:41187 | HEALTHY ca720a15d16d4077aa7ae6218d6fbce5 | 127.30.119.65:50109 | HEALTHY Flag | Value | Tags | Tablet Server ------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+------------------------- ipki_server_key_size | 1024 | experimental | all 6 server(s) checked local_ip_for_outbound_sockets | 127.30.119.65 | experimental | 127.30.119.65:50109 local_ip_for_outbound_sockets | 127.30.119.66 | experimental | 127.30.119.66:46821 local_ip_for_outbound_sockets | 127.30.119.67 | experimental | 127.30.119.67:58765 local_ip_for_outbound_sockets | 127.30.119.68 | experimental | 127.30.119.68:51058 local_ip_for_outbound_sockets | 127.30.119.69 | experimental | 127.30.119.69:41187 local_ip_for_outbound_sockets | 127.30.119.70 | experimental | 127.30.119.70:44157 never_fsync | true | unsafe,advanced | all 6 server(s) checked raft_prepare_replacement_before_eviction | true | experimental,advanced | all 6 server(s) checked rpc_server_allow_ephemeral_ports | true | unsafe | all 6 server(s) checked server_dump_info_format | pb | hidden | all 6 server(s) checked server_dump_info_path | /data/somelongdirectorytoavoidrpathissues/src/kudutest/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1539775637668737-31197/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.30.119.65:50109 server_dump_info_path | /data/somelongdirectorytoavoidrpathissues/src/kudutest/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1539775637668737-31197/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.30.119.66:46821 server_dump_info_path | /data/somelongdirectorytoavoidrpathissues/src/kudutest/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1539775637668737-31197/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.30.119.67:58765 server_dump_info_path | /data/somelongdirectorytoavoidrpathissues/src/kudutest/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1539775637668737-31197/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.30.119.68:51058 server_dump_info_path | /data/somelongdirectorytoavoidrpathissues/src/kudutest/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1539775637668737-31197/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.30.119.69:41187 server_dump_info_path | /data/somelongdirectorytoavoidrpathissues/src/kudutest/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1539775637668737-31197/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.30.119.70:44157 tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | all 6 server(s) checked Version Summary Version | Servers ----------------+------------------------- 1.9.0-SNAPSHOT | all 7 server(s) checked Summary by table Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable ----------------------+----+---------+---------------+---------+------------+------------------+------------- RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0 TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0 | Total Count ----------------+------------- Masters | 1 Tablet Servers | 6 Tables | 2 Tablets | 21 Replicas | 63 ================== Warnings: ================== Some masters have unsafe, experimental, or hidden flags set Some tablet servers have unsafe, experimental, or hidden flags set OK I1017 04:28:37.549263 31197 raft_consensus_stress-itest.cc:268] completed iteration 2 E1017 04:28:49.205351 30356 consensus_queue.cc:699] T 26fe0fdb98f64980acd487056a22149f P ca720a15d16d4077aa7ae6218d6fbce5 [LEADER]: Error trying to read ahead of the log while preparing peer request: Incomplete: Op with index 78 is ahead of the local log (next sequential op: 76). Destination peer: Peer: permanent_uuid: "88847ecc753c428b8c75790b08f9e33f" member_type: VOTER last_known_addr { host: "127.30.119.69" port: 41187 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.74, Next index: 79, Last known committed idx: 78, Time since last communication: 0.083s E1017 04:28:49.205173 30360 consensus_queue.cc:699] T 8b03926386a44d589f148dbeb73e17ca P ca720a15d16d4077aa7ae6218d6fbce5 [LEADER]: Error trying to read ahead of the log while preparing peer request: Incomplete: Op with index 85 is ahead of the local log (next sequential op: 82). Destination peer: Peer: permanent_uuid: "88847ecc753c428b8c75790b08f9e33f" member_type: VOTER last_known_addr { host: "127.30.119.69" port: 41187 } attrs { promote: false }, Status: INVALID_TERM, Last received: 9.85, Next index: 86, Last known committed idx: 85, Time since last communication: 0.086s E1017 04:28:49.214125 30377 consensus_queue.cc:699] T 9dd66394bfb145828597c3d9f214af89 P ca720a15d16d4077aa7ae6218d6fbce5 [LEADER]: Error trying to read ahead of the log while preparing peer request: Incomplete: Op with index 78 is ahead of the local log (next sequential op: 76). Destination peer: Peer: permanent_uuid: "88847ecc753c428b8c75790b08f9e33f" member_type: VOTER last_known_addr { host: "127.30.119.69" port: 41187 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.78, Next index: 79, Last known committed idx: 78, Time since last communication: 0.095s E1017 04:28:49.216852 30367 consensus_queue.cc:699] T 9dd66394bfb145828597c3d9f214af89 P ca720a15d16d4077aa7ae6218d6fbce5 [LEADER]: Error trying to read ahead of the log while preparing peer request: Incomplete: Op with index 78 is ahead of the local log (next sequential op: 76). Destination peer: Peer: permanent_uuid: "490555426e0148be8a38ccf90bedb317" member_type: VOTER last_known_addr { host: "127.30.119.66" port: 46821 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.74, Next index: 79, Last known committed idx: 78, Time since last communication: 0.009s E1017 04:28:49.229951 24989 consensus_queue.cc:699] T 9dd66394bfb145828597c3d9f214af89 P ca720a15d16d4077aa7ae6218d6fbce5 [LEADER]: Error trying to read ahead of the log while preparing peer request: Incomplete: Op with index 78 is ahead of the local log (next sequential op: 76). Destination peer: Peer: permanent_uuid: "88847ecc753c428b8c75790b08f9e33f" member_type: VOTER last_known_addr { host: "127.30.119.69" port: 41187 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.78, Next index: 79, Last known committed idx: 78, Time since last communication: 0.109s E1017 04:28:55.323055 736 consensus_queue.cc:699] T 8b03926386a44d589f148dbeb73e17ca P 3ab543a6ac3449a8bdd5c44a04ea5ee6 [LEADER]: Error trying to read ahead of the log while preparing peer request: Incomplete: Op with index 89 is ahead of the local log (next sequential op: 87). Destination peer: Peer: permanent_uuid: "88847ecc753c428b8c75790b08f9e33f" member_type: VOTER last_known_addr { host: "127.30.119.69" port: 41187 } attrs { promote: false }, Status: INVALID_TERM, Last received: 9.85, Next index: 90, Last known committed idx: 89, Time since last communication: 0.010s Master Summary UUID | Address | Status ----------------------------------+----------------------+--------- 0461054002fe43ac852199cb3befad20 | 127.30.119.126:56502 | HEALTHY Flag | Value | Tags | Master ------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+------------------------- ipki_ca_key_size | 1024 | experimental | all 1 server(s) checked ipki_server_key_size | 1024 | experimental | all 1 server(s) checked never_fsync | true | unsafe,advanced | all 1 server(s) checked raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked rpc_reuseport | true | experimental | all 1 server(s) checked rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked server_dump_info_format | pb | hidden | all 1 server(s) checked server_dump_info_path | /data/somelongdirectorytoavoidrpathissues/src/kudutest/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1539775637668737-31197/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked Tablet Server Summary UUID | Address | Status ----------------------------------+---------------------+--------- 167e75d199c246a0839f72a87f6b3416 | 127.30.119.67:58765 | HEALTHY 3ab543a6ac3449a8bdd5c44a04ea5ee6 | 127.30.119.70:44157 | HEALTHY 490555426e0148be8a38ccf90bedb317 | 127.30.119.66:46821 | HEALTHY 803373f5e35f4a61b5a983ea8b66edca | 127.30.119.68:51058 | HEALTHY 88847ecc753c428b8c75790b08f9e33f | 127.30.119.69:41187 | HEALTHY ca720a15d16d4077aa7ae6218d6fbce5 | 127.30.119.65:50109 | HEALTHY Flag | Value | Tags | Tablet Server ------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+------------------------- ipki_server_key_size | 1024 | experimental | all 6 server(s) checked local_ip_for_outbound_sockets | 127.30.119.65 | experimental | 127.30.119.65:50109 local_ip_for_outbound_sockets | 127.30.119.66 | experimental | 127.30.119.66:46821 local_ip_for_outbound_sockets | 127.30.119.67 | experimental | 127.30.119.67:58765 local_ip_for_outbound_sockets | 127.30.119.68 | experimental | 127.30.119.68:51058 local_ip_for_outbound_sockets | 127.30.119.69 | experimental | 127.30.119.69:41187 local_ip_for_outbound_sockets | 127.30.119.70 | experimental | 127.30.119.70:44157 never_fsync | true | unsafe,advanced | all 6 server(s) checked raft_prepare_replacement_before_eviction | true | experimental,advanced | all 6 server(s) checked rpc_server_allow_ephemeral_ports | true | unsafe | all 6 server(s) checked server_dump_info_format | pb | hidden | all 6 server(s) checked server_dump_info_path | /data/somelongdirectorytoavoidrpathissues/src/kudutest/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1539775637668737-31197/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.30.119.65:50109 server_dump_info_path | /data/somelongdirectorytoavoidrpathissues/src/kudutest/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1539775637668737-31197/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.30.119.66:46821 server_dump_info_path | /data/somelongdirectorytoavoidrpathissues/src/kudutest/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1539775637668737-31197/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.30.119.67:58765 server_dump_info_path | /data/somelongdirectorytoavoidrpathissues/src/kudutest/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1539775637668737-31197/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.30.119.68:51058 server_dump_info_path | /data/somelongdirectorytoavoidrpathissues/src/kudutest/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1539775637668737-31197/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.30.119.69:41187 server_dump_info_path | /data/somelongdirectorytoavoidrpathissues/src/kudutest/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1539775637668737-31197/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.30.119.70:44157 tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | all 6 server(s) checked Version Summary Version | Servers ----------------+------------------------- 1.9.0-SNAPSHOT | all 7 server(s) checked Tablet 666c7eafe4f74185886e6c52fec59dcd of table 'RemoveReplaceInCycle' is conflicted: Tablet 666c7eafe4f74185886e6c52fec59dcd of table 'RemoveReplaceInCycle' replicas' active configs disagree with the master's 490555426e0148be8a38ccf90bedb317 (127.30.119.66:46821): RUNNING [LEADER] 803373f5e35f4a61b5a983ea8b66edca (127.30.119.68:51058): RUNNING 167e75d199c246a0839f72a87f6b3416 (127.30.119.67:58765): RUNNING All reported replicas are: A = 490555426e0148be8a38ccf90bedb317 B = 803373f5e35f4a61b5a983ea8b66edca C = 167e75d199c246a0839f72a87f6b3416 The consensus matrix is: Config source | Replicas | Current term | Config index | Committed? ---------------+--------------+--------------+--------------+------------ master | A* B C | | | Yes A | A B C | 4 | 73 | Yes B | A B C | 4 | 73 | Yes C | A* B C | 3 | 73 | Yes Summary by table Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable ----------------------+----+--------------------+---------------+---------+------------+------------------+------------- TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0 RemoveReplaceInCycle | 3 | CONSENSUS_MISMATCH | 20 | 19 | 0 | 0 | 1 | Total Count ----------------+------------- Masters | 1 Tablet Servers | 6 Tables | 2 Tablets | 21 Replicas | 63 ================== Warnings: ================== Some masters have unsafe, experimental, or hidden flags set Some tablet servers have unsafe, experimental, or hidden flags set ================== Errors: ================== Corruption: table consistency check error: 1 out of 2 table(s) are not healthy FAILED E1017 04:29:01.634078 978 consensus_queue.cc:699] T 8b03926386a44d589f148dbeb73e17ca P 88847ecc753c428b8c75790b08f9e33f [LEADER]: Error trying to read ahead of the log while preparing peer request: Incomplete: Op with index 94 is ahead of the local log (next sequential op: 92). Destination peer: Peer: permanent_uuid: "ca720a15d16d4077aa7ae6218d6fbce5" member_type: VOTER last_known_addr { host: "127.30.119.65" port: 50109 } attrs { promote: false }, Status: INVALID_TERM, Last received: 10.90, Next index: 95, Last known committed idx: 94, Time since last communication: 0.020s E1017 04:29:01.634768 979 consensus_queue.cc:699] T 26fe0fdb98f64980acd487056a22149f P 88847ecc753c428b8c75790b08f9e33f [LEADER]: Error trying to read ahead of the log while preparing peer request: Incomplete: Op with index 86 is ahead of the local log (next sequential op: 84). Destination peer: Peer: permanent_uuid: "ca720a15d16d4077aa7ae6218d6fbce5" member_type: VOTER last_known_addr { host: "127.30.119.65" port: 50109 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.86, Next index: 87, Last known committed idx: 86, Time since last communication: 0.026s Master Summary UUID | Address | Status ----------------------------------+----------------------+--------- 0461054002fe43ac852199cb3befad20 | 127.30.119.126:56502 | HEALTHY Flag | Value | Tags | Master ------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+------------------------- ipki_ca_key_size | 1024 | experimental | all 1 server(s) checked ipki_server_key_size | 1024 | experimental | all 1 server(s) checked never_fsync | true | unsafe,advanced | all 1 server(s) checked raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked rpc_reuseport | true | experimental | all 1 server(s) checked rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked server_dump_info_format | pb | hidden | all 1 server(s) checked server_dump_info_path | /data/somelongdirectorytoavoidrpathissues/src/kudutest/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1539775637668737-31197/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked Tablet Server Summary UUID | Address | Status ----------------------------------+---------------------+--------- 167e75d199c246a0839f72a87f6b3416 | 127.30.119.67:58765 | HEALTHY 3ab543a6ac3449a8bdd5c44a04ea5ee6 | 127.30.119.70:44157 | HEALTHY 490555426e0148be8a38ccf90bedb317 | 127.30.119.66:46821 | HEALTHY 803373f5e35f4a61b5a983ea8b66edca | 127.30.119.68:51058 | HEALTHY 88847ecc753c428b8c75790b08f9e33f | 127.30.119.69:41187 | HEALTHY ca720a15d16d4077aa7ae6218d6fbce5 | 127.30.119.65:50109 | HEALTHY Flag | Value | Tags | Tablet Server ------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+------------------------- ipki_server_key_size | 1024 | experimental | all 6 server(s) checked local_ip_for_outbound_sockets | 127.30.119.65 | experimental | 127.30.119.65:50109 local_ip_for_outbound_sockets | 127.30.119.66 | experimental | 127.30.119.66:46821 local_ip_for_outbound_sockets | 127.30.119.67 | experimental | 127.30.119.67:58765 local_ip_for_outbound_sockets | 127.30.119.68 | experimental | 127.30.119.68:51058 local_ip_for_outbound_sockets | 127.30.119.69 | experimental | 127.30.119.69:41187 local_ip_for_outbound_sockets | 127.30.119.70 | experimental | 127.30.119.70:44157 never_fsync | true | unsafe,advanced | all 6 server(s) checked raft_prepare_replacement_before_eviction | true | experimental,advanced | all 6 server(s) checked rpc_server_allow_ephemeral_ports | true | unsafe | all 6 server(s) checked server_dump_info_format | pb | hidden | all 6 server(s) checked server_dump_info_path | /data/somelongdirectorytoavoidrpathissues/src/kudutest/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1539775637668737-31197/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.30.119.65:50109 server_dump_info_path | /data/somelongdirectorytoavoidrpathissues/src/kudutest/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1539775637668737-31197/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.30.119.66:46821 server_dump_info_path | /data/somelongdirectorytoavoidrpathissues/src/kudutest/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1539775637668737-31197/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.30.119.67:58765 server_dump_info_path | /data/somelongdirectorytoavoidrpathissues/src/kudutest/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1539775637668737-31197/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.30.119.68:51058 server_dump_info_path | /data/somelongdirectorytoavoidrpathissues/src/kudutest/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1539775637668737-31197/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.30.119.69:41187 server_dump_info_path | /data/somelongdirectorytoavoidrpathissues/src/kudutest/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1539775637668737-31197/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.30.119.70:44157 tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | all 6 server(s) checked Version Summary Version | Servers ----------------+------------------------- 1.9.0-SNAPSHOT | all 7 server(s) checked Summary by table Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable ----------------------+----+---------+---------------+---------+------------+------------------+------------- RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0 TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0 | Total Count ----------------+------------- Masters | 1 Tablet Servers | 6 Tables | 2 Tablets | 21 Replicas | 63 ================== Warnings: ================== Some masters have unsafe, experimental, or hidden flags set Some tablet servers have unsafe, experimental, or hidden flags set OK I1017 04:29:03.083488 31197 raft_consensus_stress-itest.cc:268] completed iteration 3 E1017 04:29:14.687116 1599 consensus_queue.cc:699] T 31fde0e543f84efeb62cf97bdc602900 P 3ab543a6ac3449a8bdd5c44a04ea5ee6 [LEADER]: Error trying to read ahead of the log while preparing peer request: Incomplete: Op with index 100 is ahead of the local log (next sequential op: 98). Destination peer: Peer: permanent_uuid: "803373f5e35f4a61b5a983ea8b66edca" member_type: VOTER last_known_addr { host: "127.30.119.68" port: 51058 } attrs { promote: false }, Status: INVALID_TERM, Last received: 8.100, Next index: 101, Last known committed idx: 100, Time since last communication: 0.014s W1017 04:29:17.881842 1635 negotiation.cc:313] Failed RPC negotiation. Trace: 1017 04:29:14.881472 (+ 0us) reactor.cc:583] Submitting negotiation task for client connection to 127.30.119.65:50109 1017 04:29:14.881951 (+ 479us) negotiation.cc:98] Waiting for socket to connect 1017 04:29:14.881960 (+ 9us) client_negotiation.cc:160] Beginning negotiation 1017 04:29:14.882016 (+ 56us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 1017 04:29:17.881585 (+2999569us) negotiation.cc:304] Negotiation complete: Timed out: Client connection negotiation failed: client connection to 127.30.119.65:50109 Metrics: {"client-negotiator.queue_time_us":329} W1017 04:29:20.882623 1635 negotiation.cc:313] Failed RPC negotiation. Trace: 1017 04:29:17.882357 (+ 0us) reactor.cc:583] Submitting negotiation task for client connection to 127.30.119.65:50109 1017 04:29:17.882554 (+ 197us) negotiation.cc:98] Waiting for socket to connect 1017 04:29:17.882569 (+ 15us) client_negotiation.cc:160] Beginning negotiation 1017 04:29:17.882668 (+ 99us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 1017 04:29:20.882388 (+2999720us) negotiation.cc:304] Negotiation complete: Timed out: Client connection negotiation failed: client connection to 127.30.119.65:50109 Metrics: {"client-negotiator.queue_time_us":52} Master Summary UUID | Address | Status ----------------------------------+----------------------+--------- 0461054002fe43ac852199cb3befad20 | 127.30.119.126:56502 | HEALTHY Flag | Value | Tags | Master ------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+------------------------- ipki_ca_key_size | 1024 | experimental | all 1 server(s) checked ipki_server_key_size | 1024 | experimental | all 1 server(s) checked never_fsync | true | unsafe,advanced | all 1 server(s) checked raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked rpc_reuseport | true | experimental | all 1 server(s) checked rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked server_dump_info_format | pb | hidden | all 1 server(s) checked server_dump_info_path | /data/somelongdirectorytoavoidrpathissues/src/kudutest/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1539775637668737-31197/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked Tablet Server Summary UUID | Address | Status ----------------------------------+---------------------+------------- 167e75d199c246a0839f72a87f6b3416 | 127.30.119.67:58765 | HEALTHY 3ab543a6ac3449a8bdd5c44a04ea5ee6 | 127.30.119.70:44157 | HEALTHY 490555426e0148be8a38ccf90bedb317 | 127.30.119.66:46821 | HEALTHY 803373f5e35f4a61b5a983ea8b66edca | 127.30.119.68:51058 | HEALTHY 88847ecc753c428b8c75790b08f9e33f | 127.30.119.69:41187 | HEALTHY ca720a15d16d4077aa7ae6218d6fbce5 | 127.30.119.65:50109 | UNAVAILABLE Error from 127.30.119.65:50109: Timed out: could not get status from server: Client connection negotiation failed: client connection to 127.30.119.65:50109 (UNAVAILABLE) Flag | Value | Tags | Tablet Server ------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------------------------------------------------------------------- ipki_server_key_size | 1024 | experimental | 127.30.119.67:58765, 127.30.119.70:44157, 127.30.119.66:46821, and 2 other server(s) local_ip_for_outbound_sockets | 127.30.119.66 | experimental | 127.30.119.66:46821 local_ip_for_outbound_sockets | 127.30.119.67 | experimental | 127.30.119.67:58765 local_ip_for_outbound_sockets | 127.30.119.68 | experimental | 127.30.119.68:51058 local_ip_for_outbound_sockets | 127.30.119.69 | experimental | 127.30.119.69:41187 local_ip_for_outbound_sockets | 127.30.119.70 | experimental | 127.30.119.70:44157 never_fsync | true | unsafe,advanced | 127.30.119.67:58765, 127.30.119.70:44157, 127.30.119.66:46821, and 2 other server(s) raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.30.119.67:58765, 127.30.119.70:44157, 127.30.119.66:46821, and 2 other server(s) rpc_server_allow_ephemeral_ports | true | unsafe | 127.30.119.67:58765, 127.30.119.70:44157, 127.30.119.66:46821, and 2 other server(s) server_dump_info_format | pb | hidden | 127.30.119.67:58765, 127.30.119.70:44157, 127.30.119.66:46821, and 2 other server(s) server_dump_info_path | /data/somelongdirectorytoavoidrpathissues/src/kudutest/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1539775637668737-31197/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.30.119.66:46821 server_dump_info_path | /data/somelongdirectorytoavoidrpathissues/src/kudutest/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1539775637668737-31197/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.30.119.67:58765 server_dump_info_path | /data/somelongdirectorytoavoidrpathissues/src/kudutest/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1539775637668737-31197/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.30.119.68:51058 server_dump_info_path | /data/somelongdirectorytoavoidrpathissues/src/kudutest/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1539775637668737-31197/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.30.119.69:41187 server_dump_info_path | /data/somelongdirectorytoavoidrpathissues/src/kudutest/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1539775637668737-31197/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.30.119.70:44157 tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.30.119.67:58765, 127.30.119.70:44157, 127.30.119.66:46821, and 2 other server(s) Version Summary Version | Servers ----------------+-------------------------------------------------------------------------------------------------------------- 1.9.0-SNAPSHOT | master@127.30.119.126:56502, tserver@127.30.119.67:58765, tserver@127.30.119.70:44157, and 3 other server(s) Tablet 53b982eb47024b95984ad8754627357a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING 803373f5e35f4a61b5a983ea8b66edca (127.30.119.68:51058): RUNNING [LEADER] ca720a15d16d4077aa7ae6218d6fbce5 (127.30.119.65:50109): TS unavailable 88847ecc753c428b8c75790b08f9e33f (127.30.119.69:41187): RUNNING All reported replicas are: A = 803373f5e35f4a61b5a983ea8b66edca B = ca720a15d16d4077aa7ae6218d6fbce5 C = 88847ecc753c428b8c75790b08f9e33f The consensus matrix is: Config source | Replicas | Current term | Config index | Committed? ---------------+------------------------+--------------+--------------+------------ master | A* B C | | | Yes A | A* B C | 4 | 93 | Yes B | [config not available] | | | C | A* B C | 4 | 93 | Yes Tablet 64d62b98e0284c938ed14bad999a14e3 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING 490555426e0148be8a38ccf90bedb317 (127.30.119.66:46821): RUNNING [LEADER] 803373f5e35f4a61b5a983ea8b66edca (127.30.119.68:51058): RUNNING ca720a15d16d4077aa7ae6218d6fbce5 (127.30.119.65:50109): TS unavailable All reported replicas are: A = 490555426e0148be8a38ccf90bedb317 B = 803373f5e35f4a61b5a983ea8b66edca C = ca720a15d16d4077aa7ae6218d6fbce5 The consensus matrix is: Config source | Replicas | Current term | Config index | Committed? ---------------+------------------------+--------------+--------------+------------ master | A* B C | | | Yes A | A* B C | 7 | 86 | Yes B | A* B C | 7 | 86 | Yes C | [config not available] | | | Tablet 26fe0fdb98f64980acd487056a22149f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING 803373f5e35f4a61b5a983ea8b66edca (127.30.119.68:51058): RUNNING [LEADER] ca720a15d16d4077aa7ae6218d6fbce5 (127.30.119.65:50109): TS unavailable 490555426e0148be8a38ccf90bedb317 (127.30.119.66:46821): RUNNING All reported replicas are: A = 803373f5e35f4a61b5a983ea8b66edca B = ca720a15d16d4077aa7ae6218d6fbce5 C = 490555426e0148be8a38ccf90bedb317 The consensus matrix is: Config source | Replicas | Current term | Config index | Committed? ---------------+------------------------+--------------+--------------+------------ master | A* B C | | | Yes A | A* B C | 4 | 86 | Yes B | [config not available] | | | C | A* B C | 4 | 86 | Yes Tablet ce185e91fbc5412fa8bfb9c86df7711b of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING 490555426e0148be8a38ccf90bedb317 (127.30.119.66:46821): RUNNING [LEADER] ca720a15d16d4077aa7ae6218d6fbce5 (127.30.119.65:50109): TS unavailable 167e75d199c246a0839f72a87f6b3416 (127.30.119.67:58765): RUNNING All reported replicas are: A = 490555426e0148be8a38ccf90bedb317 B = ca720a15d16d4077aa7ae6218d6fbce5 C = 167e75d199c246a0839f72a87f6b3416 The consensus matrix is: Config source | Replicas | Current term | Config index | Committed? ---------------+------------------------+--------------+--------------+------------ master | A* B C | | | Yes A | A* B C | 6 | 99 | Yes B | [config not available] | | | C | A* B C | 6 | 99 | Yes Tablet f430bbb1fef44e65b387432c037df4dd of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING 490555426e0148be8a38ccf90bedb317 (127.30.119.66:46821): RUNNING [LEADER] ca720a15d16d4077aa7ae6218d6fbce5 (127.30.119.65:50109): TS unavailable 88847ecc753c428b8c75790b08f9e33f (127.30.119.69:41187): RUNNING All reported replicas are: A = 490555426e0148be8a38ccf90bedb317 B = ca720a15d16d4077aa7ae6218d6fbce5 C = 88847ecc753c428b8c75790b08f9e33f The consensus matrix is: Config source | Replicas | Current term | Config index | Committed? ---------------+------------------------+--------------+--------------+------------ master | A* B C | | | Yes A | A* B C | 3 | 96 | Yes B | [config not available] | | | C | A* B C | 3 | 96 | Yes Tablet bed15e9cdc6d44f69eedb4378cd2ea0f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING ca720a15d16d4077aa7ae6218d6fbce5 (127.30.119.65:50109): TS unavailable [LEADER] 88847ecc753c428b8c75790b08f9e33f (127.30.119.69:41187): RUNNING 167e75d199c246a0839f72a87f6b3416 (127.30.119.67:58765): RUNNING All reported replicas are: A = ca720a15d16d4077aa7ae6218d6fbce5 B = 88847ecc753c428b8c75790b08f9e33f C = 167e75d199c246a0839f72a87f6b3416 The consensus matrix is: Config source | Replicas | Current term | Config index | Committed? ---------------+------------------------+--------------+--------------+------------ master | A* B C | | | Yes A | [config not available] | | | B | A* B C | 10 | 109 | Yes C | A* B C | 10 | 109 | Yes Tablet 2932a59244e646d6a27beb9786b3e9b7 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING 490555426e0148be8a38ccf90bedb317 (127.30.119.66:46821): RUNNING ca720a15d16d4077aa7ae6218d6fbce5 (127.30.119.65:50109): TS unavailable [LEADER] 167e75d199c246a0839f72a87f6b3416 (127.30.119.67:58765): RUNNING All reported replicas are: A = 490555426e0148be8a38ccf90bedb317 B = ca720a15d16d4077aa7ae6218d6fbce5 C = 167e75d199c246a0839f72a87f6b3416 The consensus matrix is: Config source | Replicas | Current term | Config index | Committed? ---------------+------------------------+--------------+--------------+------------ master | A B* C | | | Yes A | A B* C | 4 | 97 | Yes B | [config not available] | | | C | A B* C | 4 | 97 | Yes Tablet 9dd66394bfb145828597c3d9f214af89 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING 490555426e0148be8a38ccf90bedb317 (127.30.119.66:46821): RUNNING [LEADER] ca720a15d16d4077aa7ae6218d6fbce5 (127.30.119.65:50109): TS unavailable 88847ecc753c428b8c75790b08f9e33f (127.30.119.69:41187): RUNNING All reported replicas are: A = 490555426e0148be8a38ccf90bedb317 B = ca720a15d16d4077aa7ae6218d6fbce5 C = 88847ecc753c428b8c75790b08f9e33f The consensus matrix is: Config source | Replicas | Current term | Config index | Committed? ---------------+------------------------+--------------+--------------+------------ master | A* B C | | | Yes A | A* B C | 5 | 88 | Yes B | [config not available] | | | C | A* B C | 5 | 88 | Yes Tablet f3233d2d28a74618a2ffcf7c41bc827d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING 490555426e0148be8a38ccf90bedb317 (127.30.119.66:46821): RUNNING [LEADER] ca720a15d16d4077aa7ae6218d6fbce5 (127.30.119.65:50109): TS unavailable 88847ecc753c428b8c75790b08f9e33f (127.30.119.69:41187): RUNNING All reported replicas are: A = 490555426e0148be8a38ccf90bedb317 B = ca720a15d16d4077aa7ae6218d6fbce5 C = 88847ecc753c428b8c75790b08f9e33f The consensus matrix is: Config source | Replicas | Current term | Config index | Committed? ---------------+------------------------+--------------+--------------+------------ master | A* B C | | | Yes A | A* B C | 5 | 89 | Yes B | [config not available] | | | C | A* B C | 5 | 89 | Yes Tablet 8b03926386a44d589f148dbeb73e17ca of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING 803373f5e35f4a61b5a983ea8b66edca (127.30.119.68:51058): RUNNING ca720a15d16d4077aa7ae6218d6fbce5 (127.30.119.65:50109): TS unavailable [LEADER] 167e75d199c246a0839f72a87f6b3416 (127.30.119.67:58765): RUNNING All reported replicas are: A = 803373f5e35f4a61b5a983ea8b66edca B = ca720a15d16d4077aa7ae6218d6fbce5 C = 167e75d199c246a0839f72a87f6b3416 The consensus matrix is: Config source | Replicas | Current term | Config index | Committed? ---------------+------------------------+--------------+--------------+------------ master | A B* C | | | Yes A | A B* C | 11 | 101 | Yes B | [config not available] | | | C | A B* C | 11 | 101 | Yes Summary by table Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable ----------------------+----+------------------+---------------+---------+------------+------------------+------------- TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0 RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 20 | 10 | 0 | 10 | 0 | Total Count ----------------+------------- Masters | 1 Tablet Servers | 6 Tables | 2 Tablets | 21 Replicas | 63 ================== Warnings: ================== Some masters have unsafe, experimental, or hidden flags set unable to get flag information for tablet server ca720a15d16d4077aa7ae6218d6fbce5 (127.30.119.65:50109): Client connection negotiation failed: client connection to 127.30.119.65:50109 Some tablet servers have unsafe, experimental, or hidden flags set tserver flag check error: 1 of 6 tservers' flags were not available ================== Errors: ================== Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors Corruption: table consistency check error: 1 out of 2 table(s) are not healthy FAILED Master Summary UUID | Address | Status ----------------------------------+----------------------+--------- 0461054002fe43ac852199cb3befad20 | 127.30.119.126:56502 | HEALTHY Flag | Value | Tags | Master ------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+------------------------- ipki_ca_key_size | 1024 | experimental | all 1 server(s) checked ipki_server_key_size | 1024 | experimental | all 1 server(s) checked never_fsync | true | unsafe,advanced | all 1 server(s) checked raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked rpc_reuseport | true | experimental | all 1 server(s) checked rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked server_dump_info_format | pb | hidden | all 1 server(s) checked server_dump_info_path | /data/somelongdirectorytoavoidrpathissues/src/kudutest/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1539775637668737-31197/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked Tablet Server Summary UUID | Address | Status ----------------------------------+---------------------+--------- 167e75d199c246a0839f72a87f6b3416 | 127.30.119.67:58765 | HEALTHY 3ab543a6ac3449a8bdd5c44a04ea5ee6 | 127.30.119.70:44157 | HEALTHY 490555426e0148be8a38ccf90bedb317 | 127.30.119.66:46821 | HEALTHY 803373f5e35f4a61b5a983ea8b66edca | 127.30.119.68:51058 | HEALTHY 88847ecc753c428b8c75790b08f9e33f | 127.30.119.69:41187 | HEALTHY ca720a15d16d4077aa7ae6218d6fbce5 | 127.30.119.65:50109 | HEALTHY Flag | Value | Tags | Tablet Server ------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+------------------------- ipki_server_key_size | 1024 | experimental | all 6 server(s) checked local_ip_for_outbound_sockets | 127.30.119.65 | experimental | 127.30.119.65:50109 local_ip_for_outbound_sockets | 127.30.119.66 | experimental | 127.30.119.66:46821 local_ip_for_outbound_sockets | 127.30.119.67 | experimental | 127.30.119.67:58765 local_ip_for_outbound_sockets | 127.30.119.68 | experimental | 127.30.119.68:51058 local_ip_for_outbound_sockets | 127.30.119.69 | experimental | 127.30.119.69:41187 local_ip_for_outbound_sockets | 127.30.119.70 | experimental | 127.30.119.70:44157 never_fsync | true | unsafe,advanced | all 6 server(s) checked raft_prepare_replacement_before_eviction | true | experimental,advanced | all 6 server(s) checked rpc_server_allow_ephemeral_ports | true | unsafe | all 6 server(s) checked server_dump_info_format | pb | hidden | all 6 server(s) checked server_dump_info_path | /data/somelongdirectorytoavoidrpathissues/src/kudutest/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1539775637668737-31197/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.30.119.65:50109 server_dump_info_path | /data/somelongdirectorytoavoidrpathissues/src/kudutest/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1539775637668737-31197/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.30.119.66:46821 server_dump_info_path | /data/somelongdirectorytoavoidrpathissues/src/kudutest/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1539775637668737-31197/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.30.119.67:58765 server_dump_info_path | /data/somelongdirectorytoavoidrpathissues/src/kudutest/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1539775637668737-31197/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.30.119.68:51058 server_dump_info_path | /data/somelongdirectorytoavoidrpathissues/src/kudutest/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1539775637668737-31197/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.30.119.69:41187 server_dump_info_path | /data/somelongdirectorytoavoidrpathissues/src/kudutest/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1539775637668737-31197/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.30.119.70:44157 tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | all 6 server(s) checked Version Summary Version | Servers ----------------+------------------------- 1.9.0-SNAPSHOT | all 7 server(s) checked Summary by table Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable ----------------------+----+---------+---------------+---------+------------+------------------+------------- RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0 TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0 | Total Count ----------------+------------- Masters | 1 Tablet Servers | 6 Tables | 2 Tablets | 21 Replicas | 63 ================== Warnings: ================== Some masters have unsafe, experimental, or hidden flags set Some tablet servers have unsafe, experimental, or hidden flags set OK I1017 04:29:22.623884 31197 raft_consensus_stress-itest.cc:268] completed iteration 4 W1017 04:29:25.679581 1972 negotiation.cc:313] Failed RPC negotiation. Trace: 1017 04:29:22.679621 (+ 0us) reactor.cc:583] Submitting negotiation task for client connection to 127.30.119.69:41187 1017 04:29:22.679753 (+ 132us) negotiation.cc:98] Waiting for socket to connect 1017 04:29:22.679763 (+ 10us) client_negotiation.cc:160] Beginning negotiation 1017 04:29:22.679830 (+ 67us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 1017 04:29:25.679386 (+2999556us) negotiation.cc:304] Negotiation complete: Timed out: Client connection negotiation failed: client connection to 127.30.119.69:41187 Metrics: {"client-negotiator.queue_time_us":35} Master Summary UUID | Address | Status ----------------------------------+----------------------+--------- 0461054002fe43ac852199cb3befad20 | 127.30.119.126:56502 | HEALTHY Flag | Value | Tags | Master ------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+------------------------- ipki_ca_key_size | 1024 | experimental | all 1 server(s) checked ipki_server_key_size | 1024 | experimental | all 1 server(s) checked never_fsync | true | unsafe,advanced | all 1 server(s) checked raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked rpc_reuseport | true | experimental | all 1 server(s) checked rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked server_dump_info_format | pb | hidden | all 1 server(s) checked server_dump_info_path | /data/somelongdirectorytoavoidrpathissues/src/kudutest/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1539775637668737-31197/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked Tablet Server Summary UUID | Address | Status ----------------------------------+---------------------+------------- 167e75d199c246a0839f72a87f6b3416 | 127.30.119.67:58765 | HEALTHY 3ab543a6ac3449a8bdd5c44a04ea5ee6 | 127.30.119.70:44157 | HEALTHY 490555426e0148be8a38ccf90bedb317 | 127.30.119.66:46821 | HEALTHY 803373f5e35f4a61b5a983ea8b66edca | 127.30.119.68:51058 | HEALTHY ca720a15d16d4077aa7ae6218d6fbce5 | 127.30.119.65:50109 | HEALTHY 88847ecc753c428b8c75790b08f9e33f | 127.30.119.69:41187 | UNAVAILABLE Error from 127.30.119.69:41187: Timed out: could not get status from server: Client connection negotiation failed: client connection to 127.30.119.69:41187 (UNAVAILABLE) Flag | Value | Tags | Tablet Server ------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+------------------------- ipki_server_key_size | 1024 | experimental | all 6 server(s) checked local_ip_for_outbound_sockets | 127.30.119.65 | experimental | 127.30.119.65:50109 local_ip_for_outbound_sockets | 127.30.119.66 | experimental | 127.30.119.66:46821 local_ip_for_outbound_sockets | 127.30.119.67 | experimental | 127.30.119.67:58765 local_ip_for_outbound_sockets | 127.30.119.68 | experimental | 127.30.119.68:51058 local_ip_for_outbound_sockets | 127.30.119.69 | experimental | 127.30.119.69:41187 local_ip_for_outbound_sockets | 127.30.119.70 | experimental | 127.30.119.70:44157 never_fsync | true | unsafe,advanced | all 6 server(s) checked raft_prepare_replacement_before_eviction | true | experimental,advanced | all 6 server(s) checked rpc_server_allow_ephemeral_ports | true | unsafe | all 6 server(s) checked server_dump_info_format | pb | hidden | all 6 server(s) checked server_dump_info_path | /data/somelongdirectorytoavoidrpathissues/src/kudutest/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1539775637668737-31197/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.30.119.65:50109 server_dump_info_path | /data/somelongdirectorytoavoidrpathissues/src/kudutest/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1539775637668737-31197/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.30.119.66:46821 server_dump_info_path | /data/somelongdirectorytoavoidrpathissues/src/kudutest/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1539775637668737-31197/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.30.119.67:58765 server_dump_info_path | /data/somelongdirectorytoavoidrpathissues/src/kudutest/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1539775637668737-31197/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.30.119.68:51058 server_dump_info_path | /data/somelongdirectorytoavoidrpathissues/src/kudutest/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1539775637668737-31197/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.30.119.69:41187 server_dump_info_path | /data/somelongdirectorytoavoidrpathissues/src/kudutest/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1539775637668737-31197/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.30.119.70:44157 tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | all 6 server(s) checked Version Summary Version | Servers ----------------+-------------------------------------------------------------------------------------------------------------- 1.9.0-SNAPSHOT | master@127.30.119.126:56502, tserver@127.30.119.67:58765, tserver@127.30.119.70:44157, and 3 other server(s) Tablet 53b982eb47024b95984ad8754627357a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING 803373f5e35f4a61b5a983ea8b66edca (127.30.119.68:51058): RUNNING [LEADER] 88847ecc753c428b8c75790b08f9e33f (127.30.119.69:41187): TS unavailable 3ab543a6ac3449a8bdd5c44a04ea5ee6 (127.30.119.70:44157): RUNNING All reported replicas are: A = 803373f5e35f4a61b5a983ea8b66edca B = 88847ecc753c428b8c75790b08f9e33f C = 3ab543a6ac3449a8bdd5c44a04ea5ee6 The consensus matrix is: Config source | Replicas | Current term | Config index | Committed? ---------------+------------------------+--------------+--------------+------------ master | A* B C | | | Yes A | A* B C | 4 | 97 | Yes B | [config not available] | | | C | A* B C | 4 | 97 | Yes Tablet c9a30cf14a314d718ce60f43b1fa34f9 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING 803373f5e35f4a61b5a983ea8b66edca (127.30.119.68:51058): RUNNING [LEADER] 88847ecc753c428b8c75790b08f9e33f (127.30.119.69:41187): TS unavailable 167e75d199c246a0839f72a87f6b3416 (127.30.119.67:58765): RUNNING All reported replicas are: A = 803373f5e35f4a61b5a983ea8b66edca B = 88847ecc753c428b8c75790b08f9e33f C = 167e75d199c246a0839f72a87f6b3416 The consensus matrix is: Config source | Replicas | Current term | Config index | Committed? ---------------+------------------------+--------------+--------------+------------ master | A* B C | | | Yes A | A* B C | 5 | 94 | Yes B | [config not available] | | | C | A* B C | 5 | 94 | Yes Tablet 666c7eafe4f74185886e6c52fec59dcd of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING 490555426e0148be8a38ccf90bedb317 (127.30.119.66:46821): RUNNING 803373f5e35f4a61b5a983ea8b66edca (127.30.119.68:51058): RUNNING [LEADER] 88847ecc753c428b8c75790b08f9e33f (127.30.119.69:41187): TS unavailable All reported replicas are: A = 490555426e0148be8a38ccf90bedb317 B = 803373f5e35f4a61b5a983ea8b66edca C = 88847ecc753c428b8c75790b08f9e33f The consensus matrix is: Config source | Replicas | Current term | Config index | Committed? ---------------+------------------------+--------------+--------------+------------ master | A B* C | | | Yes A | A B* C | 5 | 80 | Yes B | A B* C | 5 | 80 | Yes C | [config not available] | | | Tablet 9e92fd7d71aa4588aded2ecdd67095cb of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING 490555426e0148be8a38ccf90bedb317 (127.30.119.66:46821): RUNNING 803373f5e35f4a61b5a983ea8b66edca (127.30.119.68:51058): RUNNING [LEADER] 88847ecc753c428b8c75790b08f9e33f (127.30.119.69:41187): TS unavailable All reported replicas are: A = 490555426e0148be8a38ccf90bedb317 B = 803373f5e35f4a61b5a983ea8b66edca C = 88847ecc753c428b8c75790b08f9e33f The consensus matrix is: Config source | Replicas | Current term | Config index | Committed? ---------------+------------------------+--------------+--------------+------------ master | A B* C | | | Yes A | A B* C | 3 | 89 | Yes B | A B* C | 3 | 89 | Yes C | [config not available] | | | Tablet 265861d9538a4feaa5de3c1bf3af5a1a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING 803373f5e35f4a61b5a983ea8b66edca (127.30.119.68:51058): RUNNING 490555426e0148be8a38ccf90bedb317 (127.30.119.66:46821): RUNNING [LEADER] 88847ecc753c428b8c75790b08f9e33f (127.30.119.69:41187): TS unavailable All reported replicas are: A = 803373f5e35f4a61b5a983ea8b66edca B = 490555426e0148be8a38ccf90bedb317 C = 88847ecc753c428b8c75790b08f9e33f The consensus matrix is: Config source | Replicas | Current term | Config index | Committed? ---------------+------------------------+--------------+--------------+------------ master | A B* C | | | Yes A | A B* C | 8 | 91 | Yes B | A B* C | 8 | 91 | Yes C | [config not available] | | | Tablet f430bbb1fef44e65b387432c037df4dd of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING 490555426e0148be8a38ccf90bedb317 (127.30.119.66:46821): RUNNING [LEADER] 88847ecc753c428b8c75790b08f9e33f (127.30.119.69:41187): TS unavailable 167e75d199c246a0839f72a87f6b3416 (127.30.119.67:58765): RUNNING All reported replicas are: A = 490555426e0148be8a38ccf90bedb317 B = 88847ecc753c428b8c75790b08f9e33f C = 167e75d199c246a0839f72a87f6b3416 The consensus matrix is: Config source | Replicas | Current term | Config index | Committed? ---------------+------------------------+--------------+--------------+------------ master | A* B C | | | Yes A | A* B C | 3 | 99 | Yes B | [config not available] | | | C | A* B C | 3 | 99 | Yes Tablet bed15e9cdc6d44f69eedb4378cd2ea0f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING 88847ecc753c428b8c75790b08f9e33f (127.30.119.69:41187): TS unavailable 167e75d199c246a0839f72a87f6b3416 (127.30.119.67:58765): RUNNING [LEADER] 3ab543a6ac3449a8bdd5c44a04ea5ee6 (127.30.119.70:44157): RUNNING All reported replicas are: A = 88847ecc753c428b8c75790b08f9e33f B = 167e75d199c246a0839f72a87f6b3416 C = 3ab543a6ac3449a8bdd5c44a04ea5ee6 The consensus matrix is: Config source | Replicas | Current term | Config index | Committed? ---------------+------------------------+--------------+--------------+------------ master | A B* C | | | Yes A | [config not available] | | | B | A B* C | 11 | 113 | Yes C | A B* C | 11 | 113 | Yes Tablet 9dd66394bfb145828597c3d9f214af89 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING 490555426e0148be8a38ccf90bedb317 (127.30.119.66:46821): RUNNING 88847ecc753c428b8c75790b08f9e33f (127.30.119.69:41187): TS unavailable 3ab543a6ac3449a8bdd5c44a04ea5ee6 (127.30.119.70:44157): RUNNING [LEADER] All reported replicas are: A = 490555426e0148be8a38ccf90bedb317 B = 88847ecc753c428b8c75790b08f9e33f C = 3ab543a6ac3449a8bdd5c44a04ea5ee6 The consensus matrix is: Config source | Replicas | Current term | Config index | Committed? ---------------+------------------------+--------------+--------------+------------ master | A B C* | | | Yes A | A B C* | 6 | 92 | Yes B | [config not available] | | | C | A B C* | 6 | 92 | Yes Tablet 6d4d96968edc4b73a3b0de435c985e29 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING 490555426e0148be8a38ccf90bedb317 (127.30.119.66:46821): RUNNING [LEADER] 803373f5e35f4a61b5a983ea8b66edca (127.30.119.68:51058): RUNNING 88847ecc753c428b8c75790b08f9e33f (127.30.119.69:41187): TS unavailable All reported replicas are: A = 490555426e0148be8a38ccf90bedb317 B = 803373f5e35f4a61b5a983ea8b66edca C = 88847ecc753c428b8c75790b08f9e33f The consensus matrix is: Config source | Replicas | Current term | Config index | Committed? ---------------+------------------------+--------------+--------------+------------ master | A* B C | | | Yes A | A* B C | 5 | 83 | Yes B | A* B C | 5 | 83 | Yes C | [config not available] | | | Tablet f3233d2d28a74618a2ffcf7c41bc827d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING 490555426e0148be8a38ccf90bedb317 (127.30.119.66:46821): RUNNING [LEADER] 88847ecc753c428b8c75790b08f9e33f (127.30.119.69:41187): TS unavailable 167e75d199c246a0839f72a87f6b3416 (127.30.119.67:58765): RUNNING All reported replicas are: A = 490555426e0148be8a38ccf90bedb317 B = 88847ecc753c428b8c75790b08f9e33f C = 167e75d199c246a0839f72a87f6b3416 The consensus matrix is: Config source | Replicas | Current term | Config index | Committed? ---------------+------------------------+--------------+--------------+------------ master | A* B C | | | Yes A | A* B C | 5 | 92 | Yes B | [config not available] | | | C | A* B C | 5 | 92 | Yes Tablet f6a1e2e405e3449bb0f1fd86a153b0b1 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING 803373f5e35f4a61b5a983ea8b66edca (127.30.119.68:51058): RUNNING 490555426e0148be8a38ccf90bedb317 (127.30.119.66:46821): RUNNING [LEADER] 88847ecc753c428b8c75790b08f9e33f (127.30.119.69:41187): TS unavailable All reported replicas are: A = 803373f5e35f4a61b5a983ea8b66edca B = 490555426e0148be8a38ccf90bedb317 C = 88847ecc753c428b8c75790b08f9e33f The consensus matrix is: Config source | Replicas | Current term | Config index | Committed? ---------------+------------------------+--------------+--------------+------------ master | A B* C | | | Yes A | A B* C | 7 | 89 | Yes B | A B* C | 7 | 89 | Yes C | [config not available] | | | Tablet 2bdbe6551a334e3fbf2c262caefe607e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING 803373f5e35f4a61b5a983ea8b66edca (127.30.119.68:51058): RUNNING [LEADER] 88847ecc753c428b8c75790b08f9e33f (127.30.119.69:41187): TS unavailable 167e75d199c246a0839f72a87f6b3416 (127.30.119.67:58765): RUNNING All reported replicas are: A = 803373f5e35f4a61b5a983ea8b66edca B = 88847ecc753c428b8c75790b08f9e33f C = 167e75d199c246a0839f72a87f6b3416 The consensus matrix is: Config source | Replicas | Current term | Config index | Committed? ---------------+------------------------+--------------+--------------+------------ master | A* B C | | | Yes A | A* B C | 5 | 94 | Yes B | [config not available] | | | C | A* B C | 5 | 94 | Yes Tablet 4cfb1f5eca244d988010eccdf19b6c10 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING 490555426e0148be8a38ccf90bedb317 (127.30.119.66:46821): RUNNING [LEADER] 803373f5e35f4a61b5a983ea8b66edca (127.30.119.68:51058): RUNNING 88847ecc753c428b8c75790b08f9e33f (127.30.119.69:41187): TS unavailable All reported replicas are: A = 490555426e0148be8a38ccf90bedb317 B = 803373f5e35f4a61b5a983ea8b66edca C = 88847ecc753c428b8c75790b08f9e33f The consensus matrix is: Config source | Replicas | Current term | Config index | Committed? ---------------+------------------------+--------------+--------------+------------ master | A* B C | | | Yes A | A* B C | 7 | 95 | Yes B | A* B C | 7 | 95 | Yes C | [config not available] | | | Tablet 31fde0e543f84efeb62cf97bdc602900 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING 803373f5e35f4a61b5a983ea8b66edca (127.30.119.68:51058): RUNNING 88847ecc753c428b8c75790b08f9e33f (127.30.119.69:41187): TS unavailable 167e75d199c246a0839f72a87f6b3416 (127.30.119.67:58765): RUNNING [LEADER] All reported replicas are: A = 803373f5e35f4a61b5a983ea8b66edca B = 88847ecc753c428b8c75790b08f9e33f C = 167e75d199c246a0839f72a87f6b3416 The consensus matrix is: Config source | Replicas | Current term | Config index | Committed? ---------------+------------------------+--------------+--------------+------------ master | A B C* | | | Yes A | A B C* | 9 | 100 | Yes B | [config not available] | | | C | A B C* | 9 | 100 | Yes Tablet ca302959d1d94e20af56964a27aba4a4 of table 'TestTable' is under-replicated: 1 replica(s) not RUNNING 490555426e0148be8a38ccf90bedb317 (127.30.119.66:46821): RUNNING [LEADER] 88847ecc753c428b8c75790b08f9e33f (127.30.119.69:41187): TS unavailable 167e75d199c246a0839f72a87f6b3416 (127.30.119.67:58765): RUNNING All reported replicas are: A = 490555426e0148be8a38ccf90bedb317 B = 88847ecc753c428b8c75790b08f9e33f C = 167e75d199c246a0839f72a87f6b3416 The consensus matrix is: Config source | Replicas | Current term | Config index | Committed? ---------------+------------------------+--------------+--------------+------------ master | A* B C | | | Yes A | A* B C | 5 | 41 | Yes B | [config not available] | | | C | A* B C | 5 | 41 | Yes Summary by table Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable ----------------------+----+------------------+---------------+---------+------------+------------------+------------- RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 20 | 6 | 0 | 14 | 0 TestTable | 3 | UNDER_REPLICATED | 1 | 0 | 0 | 1 | 0 | Total Count ----------------+------------- Masters | 1 Tablet Servers | 6 Tables | 2 Tablets | 21 Replicas | 63 ================== Warnings: ================== Some masters have unsafe, experimental, or hidden flags set Some tablet servers have unsafe, experimental, or hidden flags set ================== Errors: ================== Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors Corruption: table consistency check error: 2 out of 2 table(s) are not healthy FAILED I1017 04:29:27.979615 31197 cluster_verifier.cc:82] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors Master Summary UUID | Address | Status ----------------------------------+----------------------+--------- 0461054002fe43ac852199cb3befad20 | 127.30.119.126:56502 | HEALTHY Flag | Value | Tags | Master ------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+------------------------- ipki_ca_key_size | 1024 | experimental | all 1 server(s) checked ipki_server_key_size | 1024 | experimental | all 1 server(s) checked never_fsync | true | unsafe,advanced | all 1 server(s) checked raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked rpc_reuseport | true | experimental | all 1 server(s) checked rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked server_dump_info_format | pb | hidden | all 1 server(s) checked server_dump_info_path | /data/somelongdirectorytoavoidrpathissues/src/kudutest/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1539775637668737-31197/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked Tablet Server Summary UUID | Address | Status ----------------------------------+---------------------+--------- 167e75d199c246a0839f72a87f6b3416 | 127.30.119.67:58765 | HEALTHY 3ab543a6ac3449a8bdd5c44a04ea5ee6 | 127.30.119.70:44157 | HEALTHY 490555426e0148be8a38ccf90bedb317 | 127.30.119.66:46821 | HEALTHY 803373f5e35f4a61b5a983ea8b66edca | 127.30.119.68:51058 | HEALTHY 88847ecc753c428b8c75790b08f9e33f | 127.30.119.69:41187 | HEALTHY ca720a15d16d4077aa7ae6218d6fbce5 | 127.30.119.65:50109 | HEALTHY Flag | Value | Tags | Tablet Server ------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+------------------------- ipki_server_key_size | 1024 | experimental | all 6 server(s) checked local_ip_for_outbound_sockets | 127.30.119.65 | experimental | 127.30.119.65:50109 local_ip_for_outbound_sockets | 127.30.119.66 | experimental | 127.30.119.66:46821 local_ip_for_outbound_sockets | 127.30.119.67 | experimental | 127.30.119.67:58765 local_ip_for_outbound_sockets | 127.30.119.68 | experimental | 127.30.119.68:51058 local_ip_for_outbound_sockets | 127.30.119.69 | experimental | 127.30.119.69:41187 local_ip_for_outbound_sockets | 127.30.119.70 | experimental | 127.30.119.70:44157 never_fsync | true | unsafe,advanced | all 6 server(s) checked raft_prepare_replacement_before_eviction | true | experimental,advanced | all 6 server(s) checked rpc_server_allow_ephemeral_ports | true | unsafe | all 6 server(s) checked server_dump_info_format | pb | hidden | all 6 server(s) checked server_dump_info_path | /data/somelongdirectorytoavoidrpathissues/src/kudutest/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1539775637668737-31197/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.30.119.65:50109 server_dump_info_path | /data/somelongdirectorytoavoidrpathissues/src/kudutest/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1539775637668737-31197/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.30.119.66:46821 server_dump_info_path | /data/somelongdirectorytoavoidrpathissues/src/kudutest/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1539775637668737-31197/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.30.119.67:58765 server_dump_info_path | /data/somelongdirectorytoavoidrpathissues/src/kudutest/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1539775637668737-31197/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.30.119.68:51058 server_dump_info_path | /data/somelongdirectorytoavoidrpathissues/src/kudutest/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1539775637668737-31197/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.30.119.69:41187 server_dump_info_path | /data/somelongdirectorytoavoidrpathissues/src/kudutest/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1539775637668737-31197/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.30.119.70:44157 tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | all 6 server(s) checked Version Summary Version | Servers ----------------+------------------------- 1.9.0-SNAPSHOT | all 7 server(s) checked Summary by table Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable ----------------------+----+---------+---------------+---------+------------+------------------+------------- RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0 TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0 | Total Count ----------------+------------- Masters | 1 Tablet Servers | 6 Tables | 2 Tablets | 21 Replicas | 63 ================== Warnings: ================== Some masters have unsafe, experimental, or hidden flags set Some tablet servers have unsafe, experimental, or hidden flags set OK I1017 04:29:28.231626 31197 log_verifier.cc:121] Checking tablet 265861d9538a4feaa5de3c1bf3af5a1a I1017 04:29:28.319715 31197 log_verifier.cc:172] Verified matching terms for 96 ops in tablet 265861d9538a4feaa5de3c1bf3af5a1a I1017 04:29:28.320075 31197 log_verifier.cc:121] Checking tablet 26fe0fdb98f64980acd487056a22149f I1017 04:29:28.407248 31197 log_verifier.cc:172] Verified matching terms for 94 ops in tablet 26fe0fdb98f64980acd487056a22149f I1017 04:29:28.407433 31197 log_verifier.cc:121] Checking tablet 2932a59244e646d6a27beb9786b3e9b7 I1017 04:29:28.496449 31197 log_verifier.cc:172] Verified matching terms for 103 ops in tablet 2932a59244e646d6a27beb9786b3e9b7 I1017 04:29:28.496631 31197 log_verifier.cc:121] Checking tablet 2bdbe6551a334e3fbf2c262caefe607e I1017 04:29:28.581094 31197 log_verifier.cc:172] Verified matching terms for 99 ops in tablet 2bdbe6551a334e3fbf2c262caefe607e I1017 04:29:28.581320 31197 log_verifier.cc:121] Checking tablet 31fde0e543f84efeb62cf97bdc602900 I1017 04:29:28.670667 31197 log_verifier.cc:172] Verified matching terms for 107 ops in tablet 31fde0e543f84efeb62cf97bdc602900 I1017 04:29:28.670842 31197 log_verifier.cc:121] Checking tablet 4cfb1f5eca244d988010eccdf19b6c10 I1017 04:29:28.757796 31197 log_verifier.cc:172] Verified matching terms for 101 ops in tablet 4cfb1f5eca244d988010eccdf19b6c10 I1017 04:29:28.757977 31197 log_verifier.cc:121] Checking tablet 53b982eb47024b95984ad8754627357a I1017 04:29:28.842726 31197 log_verifier.cc:172] Verified matching terms for 102 ops in tablet 53b982eb47024b95984ad8754627357a I1017 04:29:28.842937 31197 log_verifier.cc:121] Checking tablet 64d62b98e0284c938ed14bad999a14e3 I1017 04:29:28.949290 31197 log_verifier.cc:172] Verified matching terms for 95 ops in tablet 64d62b98e0284c938ed14bad999a14e3 I1017 04:29:28.949458 31197 log_verifier.cc:121] Checking tablet 666c7eafe4f74185886e6c52fec59dcd I1017 04:29:29.039072 31197 log_verifier.cc:172] Verified matching terms for 86 ops in tablet 666c7eafe4f74185886e6c52fec59dcd I1017 04:29:29.039256 31197 log_verifier.cc:121] Checking tablet 6d4d96968edc4b73a3b0de435c985e29 I1017 04:29:29.124785 31197 log_verifier.cc:172] Verified matching terms for 89 ops in tablet 6d4d96968edc4b73a3b0de435c985e29 I1017 04:29:29.124949 31197 log_verifier.cc:121] Checking tablet 76428cfea0ba491fb3d4acf4b994d6a1 I1017 04:29:29.204874 31197 log_verifier.cc:172] Verified matching terms for 87 ops in tablet 76428cfea0ba491fb3d4acf4b994d6a1 I1017 04:29:29.205056 31197 log_verifier.cc:121] Checking tablet 8b03926386a44d589f148dbeb73e17ca I1017 04:29:29.296401 31197 log_verifier.cc:172] Verified matching terms for 107 ops in tablet 8b03926386a44d589f148dbeb73e17ca I1017 04:29:29.296592 31197 log_verifier.cc:121] Checking tablet 9dd66394bfb145828597c3d9f214af89 I1017 04:29:29.389845 31197 log_verifier.cc:172] Verified matching terms for 98 ops in tablet 9dd66394bfb145828597c3d9f214af89 I1017 04:29:29.390041 31197 log_verifier.cc:121] Checking tablet 9e92fd7d71aa4588aded2ecdd67095cb I1017 04:29:29.470999 31197 log_verifier.cc:172] Verified matching terms for 94 ops in tablet 9e92fd7d71aa4588aded2ecdd67095cb I1017 04:29:29.471177 31197 log_verifier.cc:121] Checking tablet bed15e9cdc6d44f69eedb4378cd2ea0f I1017 04:29:29.566875 31197 log_verifier.cc:172] Verified matching terms for 117 ops in tablet bed15e9cdc6d44f69eedb4378cd2ea0f I1017 04:29:29.567078 31197 log_verifier.cc:121] Checking tablet c9a30cf14a314d718ce60f43b1fa34f9 I1017 04:29:29.658133 31197 log_verifier.cc:172] Verified matching terms for 98 ops in tablet c9a30cf14a314d718ce60f43b1fa34f9 I1017 04:29:29.658337 31197 log_verifier.cc:121] Checking tablet ca302959d1d94e20af56964a27aba4a4 I1017 04:29:29.704309 31197 log_verifier.cc:172] Verified matching terms for 44 ops in tablet ca302959d1d94e20af56964a27aba4a4 I1017 04:29:29.704437 31197 log_verifier.cc:121] Checking tablet ce185e91fbc5412fa8bfb9c86df7711b I1017 04:29:29.798954 31197 log_verifier.cc:172] Verified matching terms for 104 ops in tablet ce185e91fbc5412fa8bfb9c86df7711b I1017 04:29:29.799146 31197 log_verifier.cc:121] Checking tablet f3233d2d28a74618a2ffcf7c41bc827d I1017 04:29:29.882233 31197 log_verifier.cc:172] Verified matching terms for 97 ops in tablet f3233d2d28a74618a2ffcf7c41bc827d I1017 04:29:29.882416 31197 log_verifier.cc:121] Checking tablet f430bbb1fef44e65b387432c037df4dd I1017 04:29:29.970209 31197 log_verifier.cc:172] Verified matching terms for 104 ops in tablet f430bbb1fef44e65b387432c037df4dd I1017 04:29:29.970396 31197 log_verifier.cc:121] Checking tablet f6a1e2e405e3449bb0f1fd86a153b0b1 I1017 04:29:30.051496 31197 log_verifier.cc:172] Verified matching terms for 96 ops in tablet f6a1e2e405e3449bb0f1fd86a153b0b1 I1017 04:29:30.500157 31197 external_mini_cluster.cc:1084] Attempting to check leaks for /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver pid 31267 I1017 04:29:31.017308 31197 external_mini_cluster.cc:1043] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver with pid 31267 I1017 04:29:31.065764 31197 external_mini_cluster.cc:1084] Attempting to check leaks for /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver pid 31400 I1017 04:29:31.383481 31197 external_mini_cluster.cc:1043] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver with pid 31400 I1017 04:29:31.429795 31197 external_mini_cluster.cc:1084] Attempting to check leaks for /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver pid 31522 ================================================================= ==31522==ERROR: LeakSanitizer: detected memory leaks Direct leak of 56 byte(s) in 1 object(s) allocated from: #0 0x531928 in operator new(unsigned long) /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/llvm-6.0.0.src/projects/compiler-rt/lib/asan/asan_new_delete.cc:92 #1 0x377b29c3c8 in std::string::_Rep::_S_create(unsigned long, unsigned long, std::allocator const&) (/usr/lib64/libstdc++.so.6+0x377b29c3c8) SUMMARY: AddressSanitizer: 56 byte(s) leaked in 1 allocation(s). ================================================================= ==31522==ERROR: LeakSanitizer: detected memory leaks Direct leak of 142 byte(s) in 3 object(s) allocated from: #0 0x531928 in operator new(unsigned long) /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/llvm-6.0.0.src/projects/compiler-rt/lib/asan/asan_new_delete.cc:92 #1 0x377b29c3c8 in std::string::_Rep::_S_create(unsigned long, unsigned long, std::allocator const&) (/usr/lib64/libstdc++.so.6+0x377b29c3c8) SUMMARY: AddressSanitizer: 142 byte(s) leaked in 3 allocation(s). ================================================================= ==31522==ERROR: LeakSanitizer: detected memory leaks Direct leak of 86 byte(s) in 2 object(s) allocated from: #0 0x531928 in operator new(unsigned long) /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/llvm-6.0.0.src/projects/compiler-rt/lib/asan/asan_new_delete.cc:92 #1 0x377b29c3c8 in std::string::_Rep::_S_create(unsigned long, unsigned long, std::allocator const&) (/usr/lib64/libstdc++.so.6+0x377b29c3c8) SUMMARY: AddressSanitizer: 86 byte(s) leaked in 2 allocation(s). W1017 04:29:32.431643 31197 external_mini_cluster.cc:1100] Unable to check leaks on /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver pid 31522: Timed out: CheckLeaks RPC to 127.30.119.67:58765 timed out after 1.000s (SENT) I1017 04:29:32.431776 31197 external_mini_cluster.cc:1043] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver with pid 31522 I1017 04:29:32.480775 31197 external_mini_cluster.cc:1084] Attempting to check leaks for /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver pid 31644 ================================================================= ==31644==ERROR: LeakSanitizer: detected memory leaks Direct leak of 56 byte(s) in 1 object(s) allocated from: #0 0x531928 in operator new(unsigned long) /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/llvm-6.0.0.src/projects/compiler-rt/lib/asan/asan_new_delete.cc:92 #1 0x377b29c3c8 in std::string::_Rep::_S_create(unsigned long, unsigned long, std::allocator const&) (/usr/lib64/libstdc++.so.6+0x377b29c3c8) SUMMARY: AddressSanitizer: 56 byte(s) leaked in 1 allocation(s). ================================================================= ==31644==ERROR: LeakSanitizer: detected memory leaks Direct leak of 99 byte(s) in 2 object(s) allocated from: #0 0x531928 in operator new(unsigned long) /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/llvm-6.0.0.src/projects/compiler-rt/lib/asan/asan_new_delete.cc:92 #1 0x377b29c3c8 in std::string::_Rep::_S_create(unsigned long, unsigned long, std::allocator const&) (/usr/lib64/libstdc++.so.6+0x377b29c3c8) SUMMARY: AddressSanitizer: 99 byte(s) leaked in 2 allocation(s). ================================================================= ==31644==ERROR: LeakSanitizer: detected memory leaks Direct leak of 43 byte(s) in 1 object(s) allocated from: #0 0x531928 in operator new(unsigned long) /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/llvm-6.0.0.src/projects/compiler-rt/lib/asan/asan_new_delete.cc:92 #1 0x377b29c3c8 in std::string::_Rep::_S_create(unsigned long, unsigned long, std::allocator const&) (/usr/lib64/libstdc++.so.6+0x377b29c3c8) W1017 04:29:33.482213 31197 external_mini_cluster.cc:1100] Unable to check leaks on /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver pid 31644: Timed out: CheckLeaks RPC to 127.30.119.68:51058 timed out after 1.000s (SENT) I1017 04:29:33.482311 31197 external_mini_cluster.cc:1043] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver with pid 31644 I1017 04:29:33.529796 31197 external_mini_cluster.cc:1084] Attempting to check leaks for /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver pid 31766 I1017 04:29:33.813697 31197 external_mini_cluster.cc:1043] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver with pid 31766 I1017 04:29:33.857406 31197 external_mini_cluster.cc:1084] Attempting to check leaks for /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver pid 31888 I1017 04:29:34.162643 31197 external_mini_cluster.cc:1043] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver with pid 31888 I1017 04:29:34.207947 31197 external_mini_cluster.cc:1084] Attempting to check leaks for /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-master pid 31207 I1017 04:29:34.416663 31197 external_mini_cluster.cc:1043] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-master with pid 31207 [ OK ] RaftConsensusStressITest.RemoveReplaceInCycle (136817 ms) [----------] 1 test from RaftConsensusStressITest (136817 ms total) [----------] Global test environment tear-down [==========] 1 test from 1 test case ran. (136817 ms total) [ PASSED ] 1 test.