I0109 07:20:09.854208 1200 tablet_server_main.cc:56] Tablet server non-default flags: --raft_heartbeat_interval_ms=50 --fs_data_dirs=/var/lib/kudu/tserver --fs_wal_dir=/var/lib/kudu/tserver --tserver_master_addrs=aserbin-kudu-jepsen-master.gce.cloudera.com:7051 --heap_profile_path=/tmp/kudu-tserver.1200 --unlock_experimental_flags=true --unlock_unsafe_flags=true --flagfile=/etc/kudu/conf/tserver.gflagfile --log_dir=/var/log/kudu --logtostderr=true Tablet server version: kudu 1.3.0-SNAPSHOT revision 3a23b058af0f4de04f8827255de405fff7bf9ee1 build type RELEASE built by jenkins at 09 Jan 2017 06:07:18 PST on aserbin-kudu-jepsen-control-node-debian8.gce.cloudera.com build id 2017-01-09_06-07-06 I0109 07:20:09.866611 1200 mem_tracker.cc:140] MemTracker: hard memory limit is 0.464404 GB I0109 07:20:09.866677 1200 mem_tracker.cc:142] MemTracker: soft memory limit is 0.278643 GB I0109 07:20:09.985488 1200 tablet_server_main.cc:63] Initializing tablet server... I0109 07:20:10.035234 1200 hybrid_clock.cc:177] HybridClock initialized. Resolution in nanos?: 1 Wait times tolerance adjustment: 1.0005 Current error: 366013 I0109 07:20:10.037230 1200 env_posix.cc:1286] Not raising process file limit of 65536; it is already as high as it can go I0109 07:20:10.037488 1200 file_cache.cc:401] Constructed file cache lbm with capacity 32768 I0109 07:20:10.039268 1200 server_base.cc:179] Could not load existing FS layout: Not found: /var/lib/kudu/tserver/instance: No such file or directory (error 2) I0109 07:20:10.039280 1200 server_base.cc:180] Creating new FS layout I0109 07:20:10.059814 1200 fs_manager.cc:362] Generated new instance metadata in path /var/lib/kudu/tserver/instance: uuid: "710c236995db46699eb9055c19ac506e" format_stamp: "Formatted at 2017-01-09 15:20:10 on aserbin-kudu-jepsen-ts-4" I0109 07:20:10.067144 1200 fs_manager.cc:251] Opened local filesystem: /var/lib/kudu/tserver uuid: "710c236995db46699eb9055c19ac506e" format_stamp: "Formatted at 2017-01-09 15:20:10 on aserbin-kudu-jepsen-ts-4" I0109 07:20:10.072185 1200 tablet_server_main.cc:66] Starting tablet server... I0109 07:20:10.084348 1200 rpc_server.cc:164] RPC server started. Bound to: 0.0.0.0:7050 I0109 07:20:10.084441 1200 webserver.cc:126] Starting webserver on 0.0.0.0:8050 I0109 07:20:10.084450 1200 webserver.cc:137] Document root disabled I0109 07:20:10.084635 1200 webserver.cc:221] Webserver started. Bound to: http://0.0.0.0:8050/ I0109 07:20:10.085016 1200 tablet_server_main.cc:69] Tablet server successfully started. W0109 07:20:10.138242 1335 heartbeater.cc:453] Failed to heartbeat to aserbin-kudu-jepsen-master.gce.cloudera.com:7051: Network error: Failed to ping master at aserbin-kudu-jepsen-master.gce.cloudera.com:7051: Client connection negotiation failed: client connection to 172.31.113.125:7051: connect: Connection refused (error 111) W0109 07:20:10.199815 1335 heartbeater.cc:453] Failed to heartbeat to aserbin-kudu-jepsen-master.gce.cloudera.com:7051: Network error: Failed to ping master at aserbin-kudu-jepsen-master.gce.cloudera.com:7051: Client connection negotiation failed: client connection to 172.31.113.125:7051: connect: Connection refused (error 111) W0109 07:20:10.250424 1335 heartbeater.cc:453] Failed to heartbeat to aserbin-kudu-jepsen-master.gce.cloudera.com:7051: Network error: Failed to ping master at aserbin-kudu-jepsen-master.gce.cloudera.com:7051: Client connection negotiation failed: client connection to 172.31.113.125:7051: connect: Connection refused (error 111) W0109 07:20:10.250485 1335 heartbeater.cc:323] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts. W0109 07:20:11.300592 1335 heartbeater.cc:453] Failed to heartbeat to aserbin-kudu-jepsen-master.gce.cloudera.com:7051: Network error: Failed to ping master at aserbin-kudu-jepsen-master.gce.cloudera.com:7051: Client connection negotiation failed: client connection to 172.31.113.125:7051: connect: Connection refused (error 111) I0109 07:20:12.352119 1335 heartbeater.cc:292] Connected to a master server at aserbin-kudu-jepsen-master.gce.cloudera.com:7051 I0109 07:20:12.352176 1335 heartbeater.cc:359] Registering TS with master... I0109 07:20:12.352253 1335 heartbeater.cc:373] Master aserbin-kudu-jepsen-master.gce.cloudera.com:7051 requested a full tablet report, sending... I0109 07:20:13.352871 1335 heartbeater.cc:365] Master aserbin-kudu-jepsen-master.gce.cloudera.com:7051 was elected leader, sending a full tablet report... I0109 07:20:16.793028 1272 tablet_service.cc:630] Processing CreateTablet for tablet 9e99b710634a44f4a948132a5a180600 (table=rw-register-1483975178870 [id=0e0bbef280744ed1a8036e11eb0aaea0]), partition=RANGE (key) PARTITION UNBOUNDED I0109 07:20:16.814775 1272 ts_tablet_manager.cc:863] Registered tablet 9e99b710634a44f4a948132a5a180600 I0109 07:20:16.815117 1355 ts_tablet_manager.cc:725] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e: Bootstrapping tablet I0109 07:20:16.815315 1355 tablet_bootstrap.cc:381] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e: Bootstrap starting. I0109 07:20:16.816308 1355 tablet_bootstrap.cc:540] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e: Time spent opening tablet: real 0.000s user 0.000s sys 0.000s I0109 07:20:16.816406 1355 tablet_bootstrap.cc:483] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e: No blocks or log segments found. Creating new log. I0109 07:20:16.816575 1355 log.cc:351] Log is configured to *not* fsync() on all Append() calls I0109 07:20:16.817041 1355 tablet_bootstrap.cc:381] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e: No bootstrap required, opened a new log I0109 07:20:16.817075 1355 ts_tablet_manager.cc:730] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.000s I0109 07:20:16.818439 1355 raft_consensus.cc:287] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "710c236995db46699eb9055c19ac506e" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-4.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "1678276a36f849c28459de6b6788965a" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-5.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "591b1a1d383d4b659d7549653f4f136e" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-2.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "c5a3e756029444b2bf66ee0044f5bc35" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-1.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "7897578c54854ba0a5af32eb2aa04704" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-3.gce.cloudera.com" port: 7050 } } I0109 07:20:16.818475 1355 raft_consensus.cc:318] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0109 07:20:16.818490 1355 raft_consensus.cc:547] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 710c236995db46699eb9055c19ac506e, State: 1, Role: FOLLOWER I0109 07:20:16.818511 1355 consensus_queue.cc:175] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER I0109 07:20:16.818666 1355 ts_tablet_manager.cc:757] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s I0109 07:20:16.859515 1358 raft_consensus.cc:410] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0109 07:20:16.859617 1358 raft_consensus.cc:2158] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [term 0 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 0.203s I0109 07:20:16.859686 1358 raft_consensus.cc:434] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "710c236995db46699eb9055c19ac506e" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-4.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "1678276a36f849c28459de6b6788965a" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-5.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "591b1a1d383d4b659d7549653f4f136e" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-2.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "c5a3e756029444b2bf66ee0044f5bc35" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-1.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "7897578c54854ba0a5af32eb2aa04704" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-3.gce.cloudera.com" port: 7050 } } I0109 07:20:17.058089 1358 leader_election.cc:215] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 1678276a36f849c28459de6b6788965a I0109 07:20:17.058480 1358 leader_election.cc:215] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 591b1a1d383d4b659d7549653f4f136e I0109 07:20:17.058626 1358 leader_election.cc:215] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer c5a3e756029444b2bf66ee0044f5bc35 I0109 07:20:17.058735 1358 leader_election.cc:215] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 7897578c54854ba0a5af32eb2aa04704 W0109 07:20:17.061221 1229 leader_election.cc:277] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7897578c54854ba0a5af32eb2aa04704: Not found: Tablet not found W0109 07:20:17.061414 1226 leader_election.cc:277] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c5a3e756029444b2bf66ee0044f5bc35: Not found: Tablet not found I0109 07:20:17.148157 1292 raft_consensus.cc:2158] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [term 0 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 0.185s I0109 07:20:17.148228 1292 raft_consensus.cc:1744] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1678276a36f849c28459de6b6788965a in term 0. I0109 07:20:17.149420 1227 leader_election.cc:347] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [CANDIDATE]: Term 1 pre-election: Vote granted by peer 1678276a36f849c28459de6b6788965a I0109 07:20:17.257100 1292 raft_consensus.cc:2158] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [term 0 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 0.151s I0109 07:20:17.257156 1292 raft_consensus.cc:1744] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 591b1a1d383d4b659d7549653f4f136e in term 0. I0109 07:20:17.258035 1229 leader_election.cc:347] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [CANDIDATE]: Term 1 pre-election: Vote granted by peer 591b1a1d383d4b659d7549653f4f136e I0109 07:20:17.258051 1229 leader_election.cc:242] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. I0109 07:20:17.258275 1363 raft_consensus.cc:2158] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [term 0 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 0.198s I0109 07:20:17.258291 1363 raft_consensus.cc:1968] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [term 0 FOLLOWER]: Leader pre-election won for term 1 I0109 07:20:17.258311 1363 raft_consensus.cc:410] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0109 07:20:17.258342 1363 raft_consensus.cc:2158] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [term 0 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 0.208s I0109 07:20:17.258347 1363 raft_consensus.cc:2202] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [term 0 FOLLOWER]: Advancing to term 1 I0109 07:20:17.258780 1363 raft_consensus.cc:434] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "710c236995db46699eb9055c19ac506e" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-4.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "1678276a36f849c28459de6b6788965a" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-5.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "591b1a1d383d4b659d7549653f4f136e" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-2.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "c5a3e756029444b2bf66ee0044f5bc35" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-1.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "7897578c54854ba0a5af32eb2aa04704" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-3.gce.cloudera.com" port: 7050 } } I0109 07:20:17.455890 1291 raft_consensus.cc:1719] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7897578c54854ba0a5af32eb2aa04704 for term 1 because replica is already servicing an update from a current leader or another vote. I0109 07:20:17.456092 1292 raft_consensus.cc:1674] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c5a3e756029444b2bf66ee0044f5bc35 in current term 1: Already voted for candidate 710c236995db46699eb9055c19ac506e in this term. I0109 07:20:17.456284 1363 leader_election.cc:215] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [CANDIDATE]: Term 1 election: Requesting vote from peer 1678276a36f849c28459de6b6788965a I0109 07:20:17.456357 1363 leader_election.cc:215] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [CANDIDATE]: Term 1 election: Requesting vote from peer 591b1a1d383d4b659d7549653f4f136e I0109 07:20:17.456432 1363 leader_election.cc:215] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [CANDIDATE]: Term 1 election: Requesting vote from peer c5a3e756029444b2bf66ee0044f5bc35 I0109 07:20:17.456516 1363 leader_election.cc:215] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [CANDIDATE]: Term 1 election: Requesting vote from peer 7897578c54854ba0a5af32eb2aa04704 I0109 07:20:17.457429 1227 leader_election.cc:347] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [CANDIDATE]: Term 1 election: Vote granted by peer 1678276a36f849c28459de6b6788965a I0109 07:20:17.457545 1226 leader_election.cc:347] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [CANDIDATE]: Term 1 election: Vote granted by peer c5a3e756029444b2bf66ee0044f5bc35 I0109 07:20:17.457562 1226 leader_election.cc:242] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0109 07:20:17.457602 1363 raft_consensus.cc:2158] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 0.187s I0109 07:20:17.457615 1363 raft_consensus.cc:1968] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [term 1 FOLLOWER]: Leader election won for term 1 I0109 07:20:17.457670 1229 leader_election.cc:361] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [CANDIDATE]: Term 1 election: Vote denied by peer 591b1a1d383d4b659d7549653f4f136e. Message: Service unavailable: T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 710c236995db46699eb9055c19ac506e for term 1 because replica is already servicing an update from a current leader or another vote. I0109 07:20:17.457820 1363 raft_consensus.cc:517] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [term 1 LEADER]: Becoming Leader. State: Replica: 710c236995db46699eb9055c19ac506e, State: 1, Role: LEADER I0109 07:20:17.457934 1363 consensus_queue.cc:157] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Current term: 1, Majority size: 3, State: 1, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "710c236995db46699eb9055c19ac506e" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-4.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "1678276a36f849c28459de6b6788965a" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-5.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "591b1a1d383d4b659d7549653f4f136e" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-2.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "c5a3e756029444b2bf66ee0044f5bc35" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-1.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "7897578c54854ba0a5af32eb2aa04704" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-3.gce.cloudera.com" port: 7050 } } I0109 07:20:17.622321 1229 leader_election.cc:361] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [CANDIDATE]: Term 1 election: Vote denied by peer 7897578c54854ba0a5af32eb2aa04704. Message: Invalid argument: T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 710c236995db46699eb9055c19ac506e in current term 1: Already voted for candidate 7897578c54854ba0a5af32eb2aa04704 in this term. I0109 07:20:17.657640 1291 raft_consensus.cc:1719] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [term 1 LEADER]: Leader election vote request: Denying vote to candidate 7897578c54854ba0a5af32eb2aa04704 for term 1 because replica is already servicing an update from a current leader or another vote. I0109 07:20:17.657788 1292 raft_consensus.cc:1705] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [term 1 LEADER]: Leader election vote request: Denying vote to candidate 591b1a1d383d4b659d7549653f4f136e for term 1 because replica is either leader or believes a valid leader to be alive. W0109 07:20:17.657840 1292 rpcz_store.cc:234] Call kudu.consensus.ConsensusService.RequestConsensusVote from 172.31.113.2:56186 (request call id 2) took 200ms (client timeout 196). W0109 07:20:17.657914 1292 rpcz_store.cc:238] Trace: 0109 07:20:17.457780 (+ 0us) service_pool.cc:143] Inserting onto call queue 0109 07:20:17.457789 (+ 9us) service_pool.cc:202] Handling call 0109 07:20:17.657827 (+200038us) spinlock_profiling.cc:255] Waited 200 ms on lock 0x3887870. stack: 0000000001a20b21 00000000009d8b77 00000000007e48af 0000000000a8971f 0000000000a8a342 0000000001a2dc19 00007f69845740a3 00007f6982e0487c ffffffffffffffff 0109 07:20:17.657831 (+ 4us) inbound_call.cc:130] Queueing success response Metrics: {"spinlock_wait_cycles":499941120} I0109 07:20:17.658859 1363 consensus_queue.cc:693] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [LEADER]: Connected to new peer: Peer: 1678276a36f849c28459de6b6788965a, Is new: false, Last received: 0.0, Next index: 1, Last known committed idx: 0, Last exchange result: ERROR, Needs tablet copy: false I0109 07:20:17.659018 1363 consensus_queue.cc:693] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [LEADER]: Connected to new peer: Peer: c5a3e756029444b2bf66ee0044f5bc35, Is new: false, Last received: 0.0, Next index: 1, Last known committed idx: 0, Last exchange result: ERROR, Needs tablet copy: false I0109 07:20:17.662428 1363 consensus_queue.cc:693] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [LEADER]: Connected to new peer: Peer: 7897578c54854ba0a5af32eb2aa04704, Is new: false, Last received: 0.0, Next index: 1, Last known committed idx: 0, Last exchange result: ERROR, Needs tablet copy: false I0109 07:20:17.805258 1292 raft_consensus.cc:1705] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate 591b1a1d383d4b659d7549653f4f136e for term 2 because replica is either leader or believes a valid leader to be alive. I0109 07:20:17.805512 1363 consensus_queue.cc:693] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [LEADER]: Connected to new peer: Peer: 591b1a1d383d4b659d7549653f4f136e, Is new: false, Last received: 0.0, Next index: 1, Last known committed idx: 0, Last exchange result: ERROR, Needs tablet copy: false W0109 07:20:23.846498 1229 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer 591b1a1d383d4b659d7549653f4f136e (aserbin-kudu-jepsen-ts-2.gce.cloudera.com:7050): Couldn't send request to peer 591b1a1d383d4b659d7549653f4f136e for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.113.2:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 1 times. W0109 07:20:23.896530 1229 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer 591b1a1d383d4b659d7549653f4f136e (aserbin-kudu-jepsen-ts-2.gce.cloudera.com:7050): Couldn't send request to peer 591b1a1d383d4b659d7549653f4f136e for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.113.2:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 2 times. W0109 07:20:23.931062 1229 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer 591b1a1d383d4b659d7549653f4f136e (aserbin-kudu-jepsen-ts-2.gce.cloudera.com:7050): Couldn't send request to peer 591b1a1d383d4b659d7549653f4f136e for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.113.2:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 3 times. W0109 07:20:23.970486 1229 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer 591b1a1d383d4b659d7549653f4f136e (aserbin-kudu-jepsen-ts-2.gce.cloudera.com:7050): Couldn't send request to peer 591b1a1d383d4b659d7549653f4f136e for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.113.2:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 4 times. W0109 07:20:23.999846 1229 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer 591b1a1d383d4b659d7549653f4f136e (aserbin-kudu-jepsen-ts-2.gce.cloudera.com:7050): Couldn't send request to peer 591b1a1d383d4b659d7549653f4f136e for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.113.2:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 5 times. W0109 07:20:24.041117 1229 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer 591b1a1d383d4b659d7549653f4f136e (aserbin-kudu-jepsen-ts-2.gce.cloudera.com:7050): Couldn't send request to peer 591b1a1d383d4b659d7549653f4f136e for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.113.2:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 6 times. W0109 07:20:24.069608 1229 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer 7897578c54854ba0a5af32eb2aa04704 (aserbin-kudu-jepsen-ts-3.gce.cloudera.com:7050): Couldn't send request to peer 7897578c54854ba0a5af32eb2aa04704 for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.114.37:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 1 times. W0109 07:20:24.089164 1229 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer 591b1a1d383d4b659d7549653f4f136e (aserbin-kudu-jepsen-ts-2.gce.cloudera.com:7050): Couldn't send request to peer 591b1a1d383d4b659d7549653f4f136e for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.113.2:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 7 times. W0109 07:20:24.119741 1229 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer 7897578c54854ba0a5af32eb2aa04704 (aserbin-kudu-jepsen-ts-3.gce.cloudera.com:7050): Couldn't send request to peer 7897578c54854ba0a5af32eb2aa04704 for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.114.37:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 2 times. W0109 07:20:24.124223 1229 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer 591b1a1d383d4b659d7549653f4f136e (aserbin-kudu-jepsen-ts-2.gce.cloudera.com:7050): Couldn't send request to peer 591b1a1d383d4b659d7549653f4f136e for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.113.2:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 8 times. W0109 07:20:24.150543 1229 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer 591b1a1d383d4b659d7549653f4f136e (aserbin-kudu-jepsen-ts-2.gce.cloudera.com:7050): Couldn't send request to peer 591b1a1d383d4b659d7549653f4f136e for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.113.2:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 9 times. W0109 07:20:24.169872 1229 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer 7897578c54854ba0a5af32eb2aa04704 (aserbin-kudu-jepsen-ts-3.gce.cloudera.com:7050): Couldn't send request to peer 7897578c54854ba0a5af32eb2aa04704 for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.114.37:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 3 times. W0109 07:20:24.175885 1229 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer 591b1a1d383d4b659d7549653f4f136e (aserbin-kudu-jepsen-ts-2.gce.cloudera.com:7050): Couldn't send request to peer 591b1a1d383d4b659d7549653f4f136e for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.113.2:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 10 times. W0109 07:20:24.210212 1229 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer 591b1a1d383d4b659d7549653f4f136e (aserbin-kudu-jepsen-ts-2.gce.cloudera.com:7050): Couldn't send request to peer 591b1a1d383d4b659d7549653f4f136e for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.113.2:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 11 times. W0109 07:20:24.219966 1229 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer 7897578c54854ba0a5af32eb2aa04704 (aserbin-kudu-jepsen-ts-3.gce.cloudera.com:7050): Couldn't send request to peer 7897578c54854ba0a5af32eb2aa04704 for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.114.37:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 4 times. W0109 07:20:24.240555 1229 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer 591b1a1d383d4b659d7549653f4f136e (aserbin-kudu-jepsen-ts-2.gce.cloudera.com:7050): Couldn't send request to peer 591b1a1d383d4b659d7549653f4f136e for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.113.2:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 12 times. I0109 07:20:30.150652 1386 tablet_server_main.cc:56] Tablet server non-default flags: --raft_heartbeat_interval_ms=50 --fs_data_dirs=/var/lib/kudu/tserver --fs_wal_dir=/var/lib/kudu/tserver --tserver_master_addrs=aserbin-kudu-jepsen-master.gce.cloudera.com:7051 --heap_profile_path=/tmp/kudu-tserver.1386 --unlock_experimental_flags=true --unlock_unsafe_flags=true --flagfile=/etc/kudu/conf/tserver.gflagfile --log_dir=/var/log/kudu --logtostderr=true Tablet server version: kudu 1.3.0-SNAPSHOT revision 3a23b058af0f4de04f8827255de405fff7bf9ee1 build type RELEASE built by jenkins at 09 Jan 2017 06:07:18 PST on aserbin-kudu-jepsen-control-node-debian8.gce.cloudera.com build id 2017-01-09_06-07-06 I0109 07:20:30.150856 1386 mem_tracker.cc:140] MemTracker: hard memory limit is 0.464404 GB I0109 07:20:30.150873 1386 mem_tracker.cc:142] MemTracker: soft memory limit is 0.278643 GB I0109 07:20:30.152091 1386 tablet_server_main.cc:63] Initializing tablet server... I0109 07:20:30.201642 1386 hybrid_clock.cc:177] HybridClock initialized. Resolution in nanos?: 1 Wait times tolerance adjustment: 1.0005 Current error: 376013 I0109 07:20:30.202527 1386 env_posix.cc:1286] Not raising process file limit of 65536; it is already as high as it can go I0109 07:20:30.202563 1386 file_cache.cc:401] Constructed file cache lbm with capacity 32768 I0109 07:20:30.203647 1386 fs_manager.cc:251] Opened local filesystem: /var/lib/kudu/tserver uuid: "710c236995db46699eb9055c19ac506e" format_stamp: "Formatted at 2017-01-09 15:20:10 on aserbin-kudu-jepsen-ts-4" I0109 07:20:30.207298 1386 ts_tablet_manager.cc:701] Loading metadata for tablet 9e99b710634a44f4a948132a5a180600 I0109 07:20:30.207586 1386 ts_tablet_manager.cc:863] Registered tablet 9e99b710634a44f4a948132a5a180600 I0109 07:20:30.207744 1400 ts_tablet_manager.cc:725] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e: Bootstrapping tablet I0109 07:20:30.207924 1400 tablet_bootstrap.cc:381] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e: Bootstrap starting. I0109 07:20:30.208235 1400 tablet_bootstrap.cc:540] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e: Time spent opening tablet: real 0.000s user 0.000s sys 0.000s I0109 07:20:30.208304 1400 tablet_bootstrap.cc:596] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e: Will attempt to recover log segment /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600/wal-000000001 to /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery/wal-000000001 I0109 07:20:30.208353 1400 tablet_bootstrap.cc:604] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e: Moving log directory /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600 to recovery directory /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery in preparation for log replay I0109 07:20:30.208590 1400 log_util.cc:315] Log segment /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery/wal-000000001 has no footer. This segment was likely being written when the server previously shut down. I0109 07:20:30.208602 1400 log_reader.cc:160] Log segment /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery/wal-000000001 was likely left in-progress after a previous crash. Will try to rebuild footer by scanning data. I0109 07:20:30.210095 1386 tablet_server_main.cc:66] Starting tablet server... I0109 07:20:30.211680 1400 log_util.cc:569] Scanning /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery/wal-000000001 for valid entry headers following offset 11219... I0109 07:20:30.286115 1400 log_util.cc:606] Found no log entry headers I0109 07:20:30.286187 1400 log_util.cc:218] Ignoring log segment corruption in /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery/wal-000000001 because there are no log entries following the corrupted one. The server probably crashed in the middle of writing an entry to the write-ahead log or downloaded an active log via tablet copy. Error detail: Corruption: CRC mismatch in log entry header: Log file corruption detected. Failed trying to read batch #0 at offset 11207 for log segment /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery/wal-000000001: Prior entries: [off=10942 REPLICATE (1.48)] [off=10976 COMMIT (1.48)] [off=11173 REPLICATE (1.49)] [off=11207 COMMIT (1.49)] I0109 07:20:30.286198 1400 log_util.cc:368] Successfully rebuilt footer for segment: /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery/wal-000000001 (valid entries through byte offset 11207) I0109 07:20:30.286262 1400 tablet.cc:983] T 9e99b710634a44f4a948132a5a180600 Rewinding schema during bootstrap to Schema [ 0:key[string NOT NULL], 1:value[int32 NOT NULL] ] I0109 07:20:30.286562 1400 log.cc:351] Log is configured to *not* fsync() on all Append() calls I0109 07:20:30.290029 1400 tablet_bootstrap.cc:381] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e: Bootstrap replayed 1/1 log segments. Stats: ops{read=49 overwritten=0 applied=49 ignored=0} inserts{seen=48 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I0109 07:20:30.290050 1400 tablet_bootstrap.cc:1017] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e: ReplayState: Previous OpId: 1.49, Committed OpId: 1.49, Pending Replicates: 0, Pending Commits: 0 I0109 07:20:30.290369 1400 tablet_bootstrap.cc:633] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e: Preparing to delete log recovery files and directory /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery I0109 07:20:30.290381 1400 tablet_bootstrap.cc:636] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e: Renaming log recovery dir from /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery to /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery-1483975230290380 I0109 07:20:30.290398 1400 tablet_bootstrap.cc:646] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e: Deleting all files from renamed log recovery directory /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery-1483975230290380 I0109 07:20:30.290500 1400 tablet_bootstrap.cc:649] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e: Completed deletion of old log recovery files and directory /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery-1483975230290380 I0109 07:20:30.290513 1400 tablet_bootstrap.cc:381] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e: Bootstrap complete. I0109 07:20:30.297883 1400 ts_tablet_manager.cc:730] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e: Time spent bootstrapping tablet: real 0.090s user 0.012s sys 0.032s I0109 07:20:30.298492 1400 raft_consensus.cc:287] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "710c236995db46699eb9055c19ac506e" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-4.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "1678276a36f849c28459de6b6788965a" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-5.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "591b1a1d383d4b659d7549653f4f136e" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-2.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "c5a3e756029444b2bf66ee0044f5bc35" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-1.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "7897578c54854ba0a5af32eb2aa04704" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-3.gce.cloudera.com" port: 7050 } } I0109 07:20:30.298521 1400 raft_consensus.cc:547] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 710c236995db46699eb9055c19ac506e, State: 1, Role: FOLLOWER I0109 07:20:30.298534 1400 consensus_queue.cc:175] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 49, Last appended: 1.49, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER I0109 07:20:30.298768 1400 ts_tablet_manager.cc:757] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s I0109 07:20:30.307003 1386 rpc_server.cc:164] RPC server started. Bound to: 0.0.0.0:7050 I0109 07:20:30.307098 1386 webserver.cc:126] Starting webserver on 0.0.0.0:8050 I0109 07:20:30.307106 1386 webserver.cc:137] Document root disabled I0109 07:20:30.307286 1386 webserver.cc:221] Webserver started. Bound to: http://0.0.0.0:8050/ I0109 07:20:30.307791 1386 tablet_server_main.cc:69] Tablet server successfully started. I0109 07:20:30.361026 1516 heartbeater.cc:292] Connected to a master server at aserbin-kudu-jepsen-master.gce.cloudera.com:7051 I0109 07:20:30.361084 1516 heartbeater.cc:359] Registering TS with master... I0109 07:20:30.361165 1516 heartbeater.cc:373] Master aserbin-kudu-jepsen-master.gce.cloudera.com:7051 requested a full tablet report, sending... I0109 07:20:30.468894 1437 raft_consensus.cc:410] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0109 07:20:30.469002 1437 raft_consensus.cc:2158] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 0.187s I0109 07:20:30.469084 1437 raft_consensus.cc:434] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "710c236995db46699eb9055c19ac506e" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-4.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "1678276a36f849c28459de6b6788965a" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-5.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "591b1a1d383d4b659d7549653f4f136e" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-2.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "c5a3e756029444b2bf66ee0044f5bc35" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-1.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "7897578c54854ba0a5af32eb2aa04704" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-3.gce.cloudera.com" port: 7050 } } I0109 07:20:30.665748 1437 leader_election.cc:215] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 1678276a36f849c28459de6b6788965a I0109 07:20:30.665832 1437 leader_election.cc:215] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 591b1a1d383d4b659d7549653f4f136e I0109 07:20:30.666399 1437 leader_election.cc:215] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer c5a3e756029444b2bf66ee0044f5bc35 I0109 07:20:30.666492 1437 leader_election.cc:215] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 7897578c54854ba0a5af32eb2aa04704 W0109 07:20:30.667201 1396 leader_election.cc:271] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1678276a36f849c28459de6b6788965a: Network error: Client connection negotiation failed: client connection to 172.31.114.39:7050: connect: Connection refused (error 111) W0109 07:20:30.667253 1398 leader_election.cc:271] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 591b1a1d383d4b659d7549653f4f136e: Network error: Client connection negotiation failed: client connection to 172.31.113.2:7050: connect: Connection refused (error 111) W0109 07:20:30.667356 1395 leader_election.cc:271] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c5a3e756029444b2bf66ee0044f5bc35: Network error: Client connection negotiation failed: client connection to 172.31.112.255:7050: connect: Connection refused (error 111) I0109 07:20:30.667369 1395 leader_election.cc:242] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. I0109 07:20:30.667583 1522 raft_consensus.cc:2158] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 0.208s I0109 07:20:30.667603 1522 raft_consensus.cc:1905] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority I0109 07:20:30.668954 1398 leader_election.cc:347] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [CANDIDATE]: Term 2 pre-election: Vote granted by peer 7897578c54854ba0a5af32eb2aa04704 I0109 07:20:31.077481 1437 raft_consensus.cc:410] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0109 07:20:31.077582 1437 raft_consensus.cc:2158] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 0.253s I0109 07:20:31.077671 1437 raft_consensus.cc:434] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "710c236995db46699eb9055c19ac506e" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-4.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "1678276a36f849c28459de6b6788965a" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-5.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "591b1a1d383d4b659d7549653f4f136e" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-2.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "c5a3e756029444b2bf66ee0044f5bc35" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-1.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "7897578c54854ba0a5af32eb2aa04704" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-3.gce.cloudera.com" port: 7050 } } I0109 07:20:31.275022 1473 raft_consensus.cc:1691] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 591b1a1d383d4b659d7549653f4f136e for term 2 because replica has last-logged OpId of term: 1 index: 49, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 46. W0109 07:20:31.275188 1473 rpcz_store.cc:234] Call kudu.consensus.ConsensusService.RequestConsensusVote from 172.31.113.2:56193 (request call id 1) took 163ms (client timeout 152). W0109 07:20:31.275238 1473 rpcz_store.cc:238] Trace: 0109 07:20:31.111333 (+ 0us) service_pool.cc:143] Inserting onto call queue 0109 07:20:31.111344 (+ 11us) service_pool.cc:202] Handling call 0109 07:20:31.275160 (+163816us) spinlock_profiling.cc:255] Waited 164 ms on lock 0x3750eb0. stack: 0000000001a20b21 00000000009d8b77 00000000007e48af 0000000000a8971f 0000000000a8a342 0000000001a2dc19 00007f5d377200a3 00007f5d35fb087c ffffffffffffffff 0109 07:20:31.275173 (+ 13us) inbound_call.cc:130] Queueing success response Metrics: {"spinlock_wait_cycles":408922880} I0109 07:20:31.275291 1472 raft_consensus.cc:1719] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7897578c54854ba0a5af32eb2aa04704 for term 2 because replica is already servicing an update from a current leader or another vote. I0109 07:20:31.275691 1437 leader_election.cc:215] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 1678276a36f849c28459de6b6788965a I0109 07:20:31.275913 1437 leader_election.cc:215] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 591b1a1d383d4b659d7549653f4f136e I0109 07:20:31.276037 1437 leader_election.cc:215] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer c5a3e756029444b2bf66ee0044f5bc35 I0109 07:20:31.276280 1437 leader_election.cc:215] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 7897578c54854ba0a5af32eb2aa04704 I0109 07:20:31.276721 1398 leader_election.cc:347] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [CANDIDATE]: Term 2 pre-election: Vote granted by peer 7897578c54854ba0a5af32eb2aa04704 I0109 07:20:31.277436 1398 leader_election.cc:347] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [CANDIDATE]: Term 2 pre-election: Vote granted by peer 591b1a1d383d4b659d7549653f4f136e I0109 07:20:31.277459 1398 leader_election.cc:242] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. I0109 07:20:31.277604 1546 raft_consensus.cc:2158] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 0.243s I0109 07:20:31.277619 1546 raft_consensus.cc:1968] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [term 1 FOLLOWER]: Leader pre-election won for term 2 I0109 07:20:31.277626 1546 raft_consensus.cc:410] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0109 07:20:31.277636 1546 raft_consensus.cc:2158] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 0.318s I0109 07:20:31.277667 1546 raft_consensus.cc:2202] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [term 1 FOLLOWER]: Advancing to term 2 I0109 07:20:31.277809 1395 leader_election.cc:347] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [CANDIDATE]: Term 2 pre-election: Vote granted by peer c5a3e756029444b2bf66ee0044f5bc35 I0109 07:20:31.277842 1396 leader_election.cc:347] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [CANDIDATE]: Term 2 pre-election: Vote granted by peer 1678276a36f849c28459de6b6788965a I0109 07:20:31.278120 1546 raft_consensus.cc:434] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "710c236995db46699eb9055c19ac506e" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-4.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "1678276a36f849c28459de6b6788965a" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-5.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "591b1a1d383d4b659d7549653f4f136e" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-2.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "c5a3e756029444b2bf66ee0044f5bc35" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-1.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "7897578c54854ba0a5af32eb2aa04704" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-3.gce.cloudera.com" port: 7050 } } I0109 07:20:31.363046 1516 heartbeater.cc:365] Master aserbin-kudu-jepsen-master.gce.cloudera.com:7051 was elected leader, sending a full tablet report... I0109 07:20:31.476541 1546 leader_election.cc:215] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [CANDIDATE]: Term 2 election: Requesting vote from peer 1678276a36f849c28459de6b6788965a I0109 07:20:31.476636 1546 leader_election.cc:215] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [CANDIDATE]: Term 2 election: Requesting vote from peer 591b1a1d383d4b659d7549653f4f136e I0109 07:20:31.476680 1546 leader_election.cc:215] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [CANDIDATE]: Term 2 election: Requesting vote from peer c5a3e756029444b2bf66ee0044f5bc35 I0109 07:20:31.476728 1546 leader_election.cc:215] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [CANDIDATE]: Term 2 election: Requesting vote from peer 7897578c54854ba0a5af32eb2aa04704 I0109 07:20:31.477674 1398 leader_election.cc:347] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [CANDIDATE]: Term 2 election: Vote granted by peer 591b1a1d383d4b659d7549653f4f136e I0109 07:20:31.477777 1396 leader_election.cc:347] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [CANDIDATE]: Term 2 election: Vote granted by peer 1678276a36f849c28459de6b6788965a I0109 07:20:31.477789 1396 leader_election.cc:242] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. I0109 07:20:31.477821 1546 raft_consensus.cc:2158] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [term 2 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 0.294s I0109 07:20:31.477835 1546 raft_consensus.cc:1968] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [term 2 FOLLOWER]: Leader election won for term 2 I0109 07:20:31.477861 1546 raft_consensus.cc:517] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [term 2 LEADER]: Becoming Leader. State: Replica: 710c236995db46699eb9055c19ac506e, State: 1, Role: LEADER I0109 07:20:31.477907 1398 leader_election.cc:347] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [CANDIDATE]: Term 2 election: Vote granted by peer 7897578c54854ba0a5af32eb2aa04704 I0109 07:20:31.477965 1395 leader_election.cc:347] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [CANDIDATE]: Term 2 election: Vote granted by peer c5a3e756029444b2bf66ee0044f5bc35 I0109 07:20:31.478077 1546 consensus_queue.cc:157] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 49, Committed index: 49, Last appended: 1.49, Current term: 2, Majority size: 3, State: 1, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "710c236995db46699eb9055c19ac506e" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-4.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "1678276a36f849c28459de6b6788965a" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-5.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "591b1a1d383d4b659d7549653f4f136e" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-2.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "c5a3e756029444b2bf66ee0044f5bc35" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-1.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "7897578c54854ba0a5af32eb2aa04704" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-3.gce.cloudera.com" port: 7050 } } W0109 07:20:36.634562 1546 net_util.cc:124] Time spent resolving address for aserbin-kudu-jepsen-ts-1.gce.cloudera.com: real 5.056s user 0.000s sys 0.000s I0109 07:20:36.684644 1462 raft_consensus.cc:1719] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [term 2 LEADER]: Leader pre-election vote request: Denying vote to candidate 591b1a1d383d4b659d7549653f4f136e for term 4 because replica is already servicing an update from a current leader or another vote. W0109 07:20:36.684773 1462 rpcz_store.cc:234] Call kudu.consensus.ConsensusService.RequestConsensusVote from 172.31.113.2:56193 (request call id 4) took 3018ms (client timeout 444). W0109 07:20:36.684813 1462 rpcz_store.cc:238] Trace: 0109 07:20:33.666554 (+ 0us) service_pool.cc:143] Inserting onto call queue 0109 07:20:33.666571 (+ 17us) service_pool.cc:202] Handling call 0109 07:20:36.684747 (+3018176us) spinlock_profiling.cc:255] Waited 3.02 s on lock 0x3750eb0. stack: 0000000001a20b21 00000000009d8c7a 00000000007e48af 0000000000a8971f 0000000000a8a342 0000000001a2dc19 00007f5d377200a3 00007f5d35fb087c ffffffffffffffff 0109 07:20:36.684758 (+ 11us) inbound_call.cc:130] Queueing success response Metrics: {"spinlock_wait_cycles":7545018240} I0109 07:20:36.685170 1461 raft_consensus.cc:1719] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [term 2 LEADER]: Leader election vote request: Denying vote to candidate 1678276a36f849c28459de6b6788965a for term 4 because replica is already servicing an update from a current leader or another vote. W0109 07:20:36.685219 1461 rpcz_store.cc:234] Call kudu.consensus.ConsensusService.RequestConsensusVote from 172.31.114.39:33574 (request call id 4) took 2820ms (client timeout 318). W0109 07:20:36.685250 1461 rpcz_store.cc:238] Trace: 0109 07:20:33.864751 (+ 0us) service_pool.cc:143] Inserting onto call queue 0109 07:20:33.864768 (+ 17us) service_pool.cc:202] Handling call 0109 07:20:36.685207 (+2820439us) spinlock_profiling.cc:255] Waited 2.82 s on lock 0x3750eb0. stack: 0000000001a20b21 00000000009d8c7a 00000000007e48af 0000000000a8971f 0000000000a8a342 0000000001a2dc19 00007f5d377200a3 00007f5d35fb087c ffffffffffffffff 0109 07:20:36.685212 (+ 5us) inbound_call.cc:130] Queueing success response Metrics: {"spinlock_wait_cycles":7050909440} I0109 07:20:36.685294 1467 raft_consensus.cc:1719] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [term 2 LEADER]: Leader pre-election vote request: Denying vote to candidate 591b1a1d383d4b659d7549653f4f136e for term 3 because replica is already servicing an update from a current leader or another vote. W0109 07:20:36.685369 1467 rpcz_store.cc:234] Call kudu.consensus.ConsensusService.RequestConsensusVote from 172.31.113.2:56193 (request call id 3) took 3860ms (client timeout 317). W0109 07:20:36.685433 1467 rpcz_store.cc:238] Trace: 0109 07:20:32.825291 (+ 0us) service_pool.cc:143] Inserting onto call queue 0109 07:20:32.825318 (+ 27us) service_pool.cc:202] Handling call 0109 07:20:36.685358 (+3860040us) spinlock_profiling.cc:255] Waited 3.86 s on lock 0x3750eb0. stack: 0000000001a20b21 00000000009d8c7a 00000000007e48af 0000000000a8971f 0000000000a8a342 0000000001a2dc19 00007f5d377200a3 00007f5d35fb087c ffffffffffffffff 0109 07:20:36.685363 (+ 5us) inbound_call.cc:130] Queueing success response Metrics: {"spinlock_wait_cycles":9649809280} I0109 07:20:36.685540 1468 raft_consensus.cc:1719] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [term 2 LEADER]: Leader pre-election vote request: Denying vote to candidate 7897578c54854ba0a5af32eb2aa04704 for term 3 because replica is already servicing an update from a current leader or another vote. W0109 07:20:36.685586 1468 rpcz_store.cc:234] Call kudu.consensus.ConsensusService.RequestConsensusVote from 172.31.114.37:47206 (request call id 3) took 4035ms (client timeout 320). W0109 07:20:36.685672 1468 rpcz_store.cc:238] Trace: 0109 07:20:32.650241 (+ 0us) service_pool.cc:143] Inserting onto call queue 0109 07:20:32.650265 (+ 24us) service_pool.cc:202] Handling call 0109 07:20:36.685575 (+4035310us) spinlock_profiling.cc:255] Waited 4.04 s on lock 0x3750eb0. stack: 0000000001a20b21 00000000009d8c7a 00000000007e48af 0000000000a8971f 0000000000a8a342 0000000001a2dc19 00007f5d377200a3 00007f5d35fb087c ffffffffffffffff 0109 07:20:36.685580 (+ 5us) inbound_call.cc:130] Queueing success response Metrics: {"spinlock_wait_cycles":10088103296} I0109 07:20:36.685703 1466 raft_consensus.cc:1719] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [term 2 LEADER]: Leader pre-election vote request: Denying vote to candidate 1678276a36f849c28459de6b6788965a for term 3 because replica is already servicing an update from a current leader or another vote. W0109 07:20:36.685750 1466 rpcz_store.cc:234] Call kudu.consensus.ConsensusService.RequestConsensusVote from 172.31.114.39:33574 (request call id 2) took 3859ms (client timeout 317). W0109 07:20:36.685773 1466 rpcz_store.cc:238] Trace: 0109 07:20:32.825784 (+ 0us) service_pool.cc:143] Inserting onto call queue 0109 07:20:32.825794 (+ 10us) service_pool.cc:202] Handling call 0109 07:20:36.685734 (+3859940us) spinlock_profiling.cc:255] Waited 3.86 s on lock 0x3750eb0. stack: 0000000001a20b21 00000000009d8c7a 00000000007e48af 0000000000a8971f 0000000000a8a342 0000000001a2dc19 00007f5d377200a3 00007f5d35fb087c ffffffffffffffff 0109 07:20:36.685738 (+ 4us) inbound_call.cc:130] Queueing success response Metrics: {"spinlock_wait_cycles":9649729792} I0109 07:20:36.685839 1465 raft_consensus.cc:1719] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [term 2 LEADER]: Leader pre-election vote request: Denying vote to candidate c5a3e756029444b2bf66ee0044f5bc35 for term 4 because replica is already servicing an update from a current leader or another vote. W0109 07:20:36.685888 1465 rpcz_store.cc:234] Call kudu.consensus.ConsensusService.RequestConsensusVote from 172.31.112.255:38008 (request call id 3) took 3850ms (client timeout 196). W0109 07:20:36.685907 1465 rpcz_store.cc:238] Trace: 0109 07:20:32.835854 (+ 0us) service_pool.cc:143] Inserting onto call queue 0109 07:20:32.835871 (+ 17us) service_pool.cc:202] Handling call 0109 07:20:36.685877 (+3850006us) spinlock_profiling.cc:255] Waited 3.85 s on lock 0x3750eb0. stack: 0000000001a20b21 00000000009d8c7a 00000000007e48af 0000000000a8971f 0000000000a8a342 0000000001a2dc19 00007f5d377200a3 00007f5d35fb087c ffffffffffffffff 0109 07:20:36.685881 (+ 4us) inbound_call.cc:130] Queueing success response Metrics: {"spinlock_wait_cycles":9624835840} I0109 07:20:36.685927 1464 raft_consensus.cc:1719] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [term 2 LEADER]: Leader pre-election vote request: Denying vote to candidate 1678276a36f849c28459de6b6788965a for term 4 because replica is already servicing an update from a current leader or another vote. W0109 07:20:36.685971 1464 rpcz_store.cc:234] Call kudu.consensus.ConsensusService.RequestConsensusVote from 172.31.114.39:33574 (request call id 3) took 3167ms (client timeout 400). W0109 07:20:36.685992 1464 rpcz_store.cc:238] Trace: 0109 07:20:33.518025 (+ 0us) service_pool.cc:143] Inserting onto call queue 0109 07:20:33.518043 (+ 18us) service_pool.cc:202] Handling call 0109 07:20:36.685962 (+3167919us) spinlock_profiling.cc:255] Waited 3.17 s on lock 0x3750eb0. stack: 0000000001a20b21 00000000009d8c7a 00000000007e48af 0000000000a8971f 0000000000a8a342 0000000001a2dc19 00007f5d377200a3 00007f5d35fb087c ffffffffffffffff 0109 07:20:36.685966 (+ 4us) inbound_call.cc:130] Queueing success response Metrics: {"spinlock_wait_cycles":7919620864} I0109 07:20:36.686012 1469 raft_consensus.cc:1719] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [term 2 LEADER]: Leader election vote request: Denying vote to candidate c5a3e756029444b2bf66ee0044f5bc35 for term 3 because replica is already servicing an update from a current leader or another vote. W0109 07:20:36.686065 1469 rpcz_store.cc:234] Call kudu.consensus.ConsensusService.RequestConsensusVote from 172.31.112.255:38008 (request call id 2) took 4047ms (client timeout 164). W0109 07:20:36.686103 1469 rpcz_store.cc:238] Trace: 0109 07:20:32.638892 (+ 0us) service_pool.cc:143] Inserting onto call queue 0109 07:20:32.638912 (+ 20us) service_pool.cc:202] Handling call 0109 07:20:36.686052 (+4047140us) spinlock_profiling.cc:255] Waited 4.05 s on lock 0x3750eb0. stack: 0000000001a20b21 00000000009d8c7a 00000000007e48af 0000000000a8971f 0000000000a8a342 0000000001a2dc19 00007f5d377200a3 00007f5d35fb087c ffffffffffffffff 0109 07:20:36.686057 (+ 5us) inbound_call.cc:130] Queueing success response Metrics: {"spinlock_wait_cycles":10117656960} I0109 07:20:36.686133 1473 raft_consensus.cc:1719] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [term 2 LEADER]: Leader pre-election vote request: Denying vote to candidate 1678276a36f849c28459de6b6788965a for term 3 because replica is already servicing an update from a current leader or another vote. W0109 07:20:36.686197 1473 rpcz_store.cc:234] Call kudu.consensus.ConsensusService.RequestConsensusVote from 172.31.114.39:33574 (request call id 1) took 4643ms (client timeout 223). W0109 07:20:36.686226 1473 rpcz_store.cc:238] Trace: 0109 07:20:32.043153 (+ 0us) service_pool.cc:143] Inserting onto call queue 0109 07:20:32.043164 (+ 11us) service_pool.cc:202] Handling call 0109 07:20:36.686173 (+4643009us) spinlock_profiling.cc:255] Waited 4.64 s on lock 0x3750eb0. stack: 0000000001a20b21 00000000009d8c7a 00000000007e48af 0000000000a8971f 0000000000a8a342 0000000001a2dc19 00007f5d377200a3 00007f5d35fb087c ffffffffffffffff 0109 07:20:36.686178 (+ 5us) inbound_call.cc:130] Queueing success response Metrics: {"spinlock_wait_cycles":11607340928} I0109 07:20:36.686251 1471 raft_consensus.cc:1719] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [term 2 LEADER]: Leader pre-election vote request: Denying vote to candidate c5a3e756029444b2bf66ee0044f5bc35 for term 3 because replica is already servicing an update from a current leader or another vote. W0109 07:20:36.686297 1471 rpcz_store.cc:234] Call kudu.consensus.ConsensusService.RequestConsensusVote from 172.31.112.255:38008 (request call id 1) took 4612ms (client timeout 221). W0109 07:20:36.686336 1471 rpcz_store.cc:238] Trace: 0109 07:20:32.073407 (+ 0us) service_pool.cc:143] Inserting onto call queue 0109 07:20:32.073420 (+ 13us) service_pool.cc:202] Handling call 0109 07:20:36.686284 (+4612864us) spinlock_profiling.cc:255] Waited 4.61 s on lock 0x3750eb0. stack: 0000000001a20b21 00000000009d8c7a 00000000007e48af 0000000000a8971f 0000000000a8a342 0000000001a2dc19 00007f5d377200a3 00007f5d35fb087c ffffffffffffffff 0109 07:20:36.686289 (+ 5us) inbound_call.cc:130] Queueing success response Metrics: {"spinlock_wait_cycles":11531998592} I0109 07:20:36.686380 1472 raft_consensus.cc:1705] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [term 2 LEADER]: Leader pre-election vote request: Denying vote to candidate 591b1a1d383d4b659d7549653f4f136e for term 3 because replica is either leader or believes a valid leader to be alive. W0109 07:20:36.686486 1472 rpcz_store.cc:234] Call kudu.consensus.ConsensusService.RequestConsensusVote from 172.31.113.2:56193 (request call id 2) took 4651ms (client timeout 235). W0109 07:20:36.686522 1472 rpcz_store.cc:238] Trace: 0109 07:20:32.034614 (+ 0us) service_pool.cc:143] Inserting onto call queue 0109 07:20:32.034644 (+ 30us) service_pool.cc:202] Handling call 0109 07:20:36.686451 (+4651807us) spinlock_profiling.cc:255] Waited 4.65 s on lock 0x3750eb0. stack: 0000000001a20b21 00000000009d8b77 00000000007e48af 0000000000a8971f 0000000000a8a342 0000000001a2dc19 00007f5d377200a3 00007f5d35fb087c ffffffffffffffff 0109 07:20:36.686476 (+ 25us) inbound_call.cc:130] Queueing success response Metrics: {"spinlock_wait_cycles":11629234944} I0109 07:20:36.686604 1459 raft_consensus.cc:2197] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [term 2 LEADER]: Stepping down as leader of term 2 I0109 07:20:36.686619 1459 raft_consensus.cc:547] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 710c236995db46699eb9055c19ac506e, State: 1, Role: LEADER I0109 07:20:36.686642 1459 consensus_queue.cc:175] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 49, Committed index: 49, Last appended: 2.50, Current term: 2, Majority size: -1, State: 1, Mode: NON_LEADER I0109 07:20:36.686934 1459 consensus_peers.cc:380] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer c5a3e756029444b2bf66ee0044f5bc35 (aserbin-kudu-jepsen-ts-1.gce.cloudera.com:7050): Closing peer: c5a3e756029444b2bf66ee0044f5bc35 I0109 07:20:36.686975 1546 consensus_queue.cc:703] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [NON_LEADER]: Peer responded invalid term: Peer: 591b1a1d383d4b659d7549653f4f136e, Is new: false, Last received: 4.54, Next index: 55, Last known committed idx: 54, Last exchange result: ERROR, Needs tablet copy: false I0109 07:20:36.687084 1459 consensus_peers.cc:380] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer 7897578c54854ba0a5af32eb2aa04704 (aserbin-kudu-jepsen-ts-3.gce.cloudera.com:7050): Closing peer: 7897578c54854ba0a5af32eb2aa04704 I0109 07:20:36.687216 1546 consensus_queue.cc:679] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [NON_LEADER]: Peer 1678276a36f849c28459de6b6788965a log is divergent from this leader: its last log entry 4.54 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 54 I0109 07:20:36.687245 1546 consensus_queue.cc:703] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [NON_LEADER]: Peer responded invalid term: Peer: 1678276a36f849c28459de6b6788965a, Is new: false, Last received: 0.0, Next index: 55, Last known committed idx: 54, Last exchange result: ERROR, Needs tablet copy: false I0109 07:20:36.687322 1459 consensus_peers.cc:380] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer 591b1a1d383d4b659d7549653f4f136e (aserbin-kudu-jepsen-ts-2.gce.cloudera.com:7050): Closing peer: 591b1a1d383d4b659d7549653f4f136e I0109 07:20:36.687383 1459 consensus_peers.cc:380] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer 1678276a36f849c28459de6b6788965a (aserbin-kudu-jepsen-ts-5.gce.cloudera.com:7050): Closing peer: 1678276a36f849c28459de6b6788965a I0109 07:20:36.687423 1459 raft_consensus.cc:2202] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [term 2 FOLLOWER]: Advancing to term 4 I0109 07:20:36.687933 1459 raft_consensus.cc:915] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [term 4 FOLLOWER]: Refusing update from remote peer 1678276a36f849c28459de6b6788965a: Log matching property violated. Preceding OpId in replica: term: 2 index: 50. Preceding OpId from leader: term: 4 index: 52. (index mismatch) W0109 07:20:36.688004 1459 rpcz_store.cc:234] Call kudu.consensus.ConsensusService.UpdateConsensus from 172.31.114.39:33574 (request call id 6) took 1531ms (client timeout 1000). W0109 07:20:36.688030 1459 rpcz_store.cc:238] Trace: 0109 07:20:35.156340 (+ 0us) service_pool.cc:143] Inserting onto call queue 0109 07:20:35.156364 (+ 24us) service_pool.cc:202] Handling call 0109 07:20:36.686589 (+1530225us) raft_consensus.cc:1123] Updating replica for 0 ops 0109 07:20:36.687947 (+ 1358us) raft_consensus.cc:1329] Filling consensus response to leader. 0109 07:20:36.687991 (+ 44us) spinlock_profiling.cc:255] Waited 1.53 s on lock 0x35f8f68. stack: 0000000001a20b21 00000000009d8510 00000000007e430a 0000000000a8971f 0000000000a8a342 0000000001a2dc19 00007f5d377200a3 00007f5d35fb087c ffffffffffffffff 0109 07:20:36.687997 (+ 6us) inbound_call.cc:130] Queueing success response Metrics: {"spinlock_wait_cycles":3825852416} I0109 07:20:36.688066 1463 raft_consensus.cc:1719] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7897578c54854ba0a5af32eb2aa04704 for term 4 because replica is already servicing an update from a current leader or another vote. W0109 07:20:36.688133 1463 rpcz_store.cc:234] Call kudu.consensus.ConsensusService.RequestConsensusVote from 172.31.114.37:47206 (request call id 4) took 3034ms (client timeout 378). W0109 07:20:36.688169 1463 rpcz_store.cc:238] Trace: 0109 07:20:33.654063 (+ 0us) service_pool.cc:143] Inserting onto call queue 0109 07:20:33.654080 (+ 17us) service_pool.cc:202] Handling call 0109 07:20:36.688110 (+3034030us) spinlock_profiling.cc:255] Waited 3.03 s on lock 0x3750eb0. stack: 0000000001a20b21 00000000009d8c7a 00000000007e48af 0000000000a8971f 0000000000a8a342 0000000001a2dc19 00007f5d377200a3 00007f5d35fb087c ffffffffffffffff 0109 07:20:36.688124 (+ 14us) inbound_call.cc:130] Queueing success response Metrics: {"spinlock_wait_cycles":7584865024} I0109 07:20:36.688225 1458 raft_consensus.cc:915] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [term 4 FOLLOWER]: Refusing update from remote peer 1678276a36f849c28459de6b6788965a: Log matching property violated. Preceding OpId in replica: term: 2 index: 50. Preceding OpId from leader: term: 4 index: 54. (index mismatch) I0109 07:20:36.688395 1470 raft_consensus.cc:1719] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7897578c54854ba0a5af32eb2aa04704 for term 3 because replica is already servicing an update from a current leader or another vote. W0109 07:20:36.688489 1470 rpcz_store.cc:234] Call kudu.consensus.ConsensusService.RequestConsensusVote from 172.31.114.37:47206 (request call id 2) took 4515ms (client timeout 303). W0109 07:20:36.688549 1470 rpcz_store.cc:238] Trace: 0109 07:20:32.172871 (+ 0us) service_pool.cc:143] Inserting onto call queue 0109 07:20:32.172892 (+ 21us) service_pool.cc:202] Handling call 0109 07:20:36.688439 (+4515547us) spinlock_profiling.cc:255] Waited 4.52 s on lock 0x3750eb0. stack: 0000000001a20b21 00000000009d8c7a 00000000007e48af 0000000000a8971f 0000000000a8a342 0000000001a2dc19 00007f5d377200a3 00007f5d35fb087c ffffffffffffffff 0109 07:20:36.688444 (+ 5us) inbound_call.cc:130] Queueing success response Metrics: {"spinlock_wait_cycles":11288657664} I0109 07:20:36.688611 1460 raft_consensus.cc:915] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [term 4 FOLLOWER]: Refusing update from remote peer 1678276a36f849c28459de6b6788965a: Log matching property violated. Preceding OpId in replica: term: 2 index: 50. Preceding OpId from leader: term: 4 index: 50. (term mismatch) I0109 07:20:36.688637 1460 raft_consensus_state.cc:382] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e: Aborting all transactions after (but not including) 49 I0109 07:20:36.688659 1460 raft_consensus_state.cc:403] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e: Aborting uncommitted NO_OP operation due to leader change: 2.50 I0109 07:20:36.688707 1460 raft_consensus.cc:2038] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e: NO_OP replication failed: Aborted: Transaction aborted by new leader W0109 07:20:36.688771 1460 rpcz_store.cc:234] Call kudu.consensus.ConsensusService.UpdateConsensus from 172.31.114.39:33574 (request call id 5) took 2582ms (client timeout 1000). W0109 07:20:36.688814 1460 rpcz_store.cc:238] Trace: 0109 07:20:34.106095 (+ 0us) service_pool.cc:143] Inserting onto call queue 0109 07:20:34.106114 (+ 19us) service_pool.cc:202] Handling call 0109 07:20:36.688581 (+2582467us) raft_consensus.cc:1123] Updating replica for 0 ops 0109 07:20:36.688733 (+ 152us) raft_consensus.cc:1329] Filling consensus response to leader. 0109 07:20:36.688757 (+ 24us) spinlock_profiling.cc:255] Waited 2.58 s on lock 0x35f8f68. stack: 0000000001a20b21 00000000009d8510 00000000007e430a 0000000000a8971f 0000000000a8a342 0000000001a2dc19 00007f5d377200a3 00007f5d35fb087c ffffffffffffffff 0109 07:20:36.688763 (+ 6us) inbound_call.cc:130] Queueing success response Metrics: {"spinlock_wait_cycles":6456080128} I0109 07:20:36.689527 1552 rpcz_store.cc:248] Call kudu.tserver.TabletServerService.Write from 172.31.112.10:39863 (ReqId={client: 551b04fdc262441283af896460477398, seq_no=11, attempt_no=1}) took 1272ms. Request Metrics: {"child_traces":[["txn",{"num_ops":1,"prepare.queue_time_us":4306,"spinlock_wait_cycles":3169285376,"thread_start_us":2067,"threads_started":1}]]} I0109 07:20:36.689666 1552 rpcz_store.cc:248] Call kudu.tserver.TabletServerService.Write from 172.31.112.10:39879 (ReqId={client: e62b2d71350446dbba6236513fdab48e, seq_no=12, attempt_no=14}) took 5011ms. Request Metrics: {"child_traces":[["txn",{"mutex_wait_us":1453,"num_ops":1,"prepare.queue_time_us":3808,"spinlock_wait_cycles":12518651264}]]} I0109 07:20:36.689782 1552 rpcz_store.cc:248] Call kudu.tserver.TabletServerService.Write from 172.31.112.10:39858 (ReqId={client: 16f99eb378094d8cb6a098a0b18f22d9, seq_no=11, attempt_no=1}) took 3235ms. Request Metrics: {"child_traces":[["txn",{"mutex_wait_us":1448,"num_ops":1,"prepare.queue_time_us":3899,"spinlock_wait_cycles":8079172480}]]} I0109 07:20:36.689926 1552 rpcz_store.cc:248] Call kudu.tserver.TabletServerService.Write from 172.31.112.10:39860 (ReqId={client: 9951e7a73b924fcca925354c6132dce4, seq_no=8, attempt_no=16}) took 2766ms. Request Metrics: {"child_traces":[["txn",{"mutex_wait_us":736,"num_ops":1,"prepare.queue_time_us":3291,"spinlock_wait_cycles":6907786624}]]} I0109 07:20:37.459544 1437 raft_consensus.cc:410] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1678276a36f849c28459de6b6788965a) I0109 07:20:37.459636 1437 raft_consensus.cc:2158] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [term 4 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 0.207s I0109 07:20:37.459712 1437 raft_consensus.cc:434] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "710c236995db46699eb9055c19ac506e" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-4.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "1678276a36f849c28459de6b6788965a" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-5.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "591b1a1d383d4b659d7549653f4f136e" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-2.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "c5a3e756029444b2bf66ee0044f5bc35" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-1.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "7897578c54854ba0a5af32eb2aa04704" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-3.gce.cloudera.com" port: 7050 } } I0109 07:20:37.657171 1437 leader_election.cc:215] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [CANDIDATE]: Term 5 pre-election: Requesting pre-vote from peer 1678276a36f849c28459de6b6788965a I0109 07:20:37.657249 1437 leader_election.cc:215] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [CANDIDATE]: Term 5 pre-election: Requesting pre-vote from peer 591b1a1d383d4b659d7549653f4f136e I0109 07:20:37.657266 1437 leader_election.cc:215] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [CANDIDATE]: Term 5 pre-election: Requesting pre-vote from peer c5a3e756029444b2bf66ee0044f5bc35 I0109 07:20:37.657285 1437 leader_election.cc:215] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [CANDIDATE]: Term 5 pre-election: Requesting pre-vote from peer 7897578c54854ba0a5af32eb2aa04704 I0109 07:20:37.657559 1460 raft_consensus.cc:2158] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [term 4 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 0.204s I0109 07:20:37.657579 1460 raft_consensus.cc:1744] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7897578c54854ba0a5af32eb2aa04704 in term 4. W0109 07:20:37.658318 1396 leader_election.cc:271] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 1678276a36f849c28459de6b6788965a: Network error: Client connection negotiation failed: client connection to 172.31.114.39:7050: connect: Connection refused (error 111) W0109 07:20:37.658428 1398 leader_election.cc:271] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 591b1a1d383d4b659d7549653f4f136e: Network error: Client connection negotiation failed: client connection to 172.31.113.2:7050: connect: Connection refused (error 111) I0109 07:20:37.658476 1398 leader_election.cc:347] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [CANDIDATE]: Term 5 pre-election: Vote granted by peer 7897578c54854ba0a5af32eb2aa04704 W0109 07:20:37.865218 1395 leader_election.cc:271] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer c5a3e756029444b2bf66ee0044f5bc35: Timed out: RequestConsensusVote RPC to 172.31.112.255:7050 timed out after 0.207s (SENT) I0109 07:20:37.865288 1395 leader_election.cc:242] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. I0109 07:20:37.865537 1559 raft_consensus.cc:2158] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [term 4 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 0.161s I0109 07:20:37.865557 1559 raft_consensus.cc:1905] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority I0109 07:20:44.106066 1569 tablet_server_main.cc:56] Tablet server non-default flags: --raft_heartbeat_interval_ms=50 --fs_data_dirs=/var/lib/kudu/tserver --fs_wal_dir=/var/lib/kudu/tserver --tserver_master_addrs=aserbin-kudu-jepsen-master.gce.cloudera.com:7051 --heap_profile_path=/tmp/kudu-tserver.1569 --unlock_experimental_flags=true --unlock_unsafe_flags=true --flagfile=/etc/kudu/conf/tserver.gflagfile --log_dir=/var/log/kudu --logtostderr=true Tablet server version: kudu 1.3.0-SNAPSHOT revision 3a23b058af0f4de04f8827255de405fff7bf9ee1 build type RELEASE built by jenkins at 09 Jan 2017 06:07:18 PST on aserbin-kudu-jepsen-control-node-debian8.gce.cloudera.com build id 2017-01-09_06-07-06 I0109 07:20:44.106314 1569 mem_tracker.cc:140] MemTracker: hard memory limit is 0.464404 GB I0109 07:20:44.106328 1569 mem_tracker.cc:142] MemTracker: soft memory limit is 0.278643 GB I0109 07:20:44.107971 1569 tablet_server_main.cc:63] Initializing tablet server... I0109 07:20:44.157589 1569 hybrid_clock.cc:177] HybridClock initialized. Resolution in nanos?: 1 Wait times tolerance adjustment: 1.0005 Current error: 383013 I0109 07:20:44.158805 1569 env_posix.cc:1286] Not raising process file limit of 65536; it is already as high as it can go I0109 07:20:44.158840 1569 file_cache.cc:401] Constructed file cache lbm with capacity 32768 I0109 07:20:44.160142 1569 fs_manager.cc:251] Opened local filesystem: /var/lib/kudu/tserver uuid: "710c236995db46699eb9055c19ac506e" format_stamp: "Formatted at 2017-01-09 15:20:10 on aserbin-kudu-jepsen-ts-4" I0109 07:20:44.164176 1569 ts_tablet_manager.cc:701] Loading metadata for tablet 9e99b710634a44f4a948132a5a180600 I0109 07:20:44.164505 1569 ts_tablet_manager.cc:863] Registered tablet 9e99b710634a44f4a948132a5a180600 I0109 07:20:44.164701 1583 ts_tablet_manager.cc:725] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e: Bootstrapping tablet I0109 07:20:44.164968 1583 tablet_bootstrap.cc:381] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e: Bootstrap starting. I0109 07:20:44.165269 1583 tablet_bootstrap.cc:540] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e: Time spent opening tablet: real 0.000s user 0.000s sys 0.000s I0109 07:20:44.165357 1583 tablet_bootstrap.cc:596] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e: Will attempt to recover log segment /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600/wal-000000001 to /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery/wal-000000001 I0109 07:20:44.165390 1583 tablet_bootstrap.cc:604] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e: Moving log directory /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600 to recovery directory /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery in preparation for log replay I0109 07:20:44.165606 1583 log_util.cc:315] Log segment /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery/wal-000000001 has no footer. This segment was likely being written when the server previously shut down. I0109 07:20:44.165635 1583 log_reader.cc:160] Log segment /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery/wal-000000001 was likely left in-progress after a previous crash. Will try to rebuild footer by scanning data. I0109 07:20:44.167049 1569 tablet_server_main.cc:66] Starting tablet server... I0109 07:20:44.169689 1583 log_util.cc:569] Scanning /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery/wal-000000001 for valid entry headers following offset 13814... I0109 07:20:44.246513 1583 log_util.cc:606] Found no log entry headers I0109 07:20:44.246587 1583 log_util.cc:218] Ignoring log segment corruption in /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery/wal-000000001 because there are no log entries following the corrupted one. The server probably crashed in the middle of writing an entry to the write-ahead log or downloaded an active log via tablet copy. Error detail: Corruption: CRC mismatch in log entry header: Log file corruption detected. Failed trying to read batch #0 at offset 13802 for log segment /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery/wal-000000001: Prior entries: [off=13537 REPLICATE (4.60)] [off=13571 COMMIT (4.60)] [off=13768 REPLICATE (4.61)] [off=13802 COMMIT (4.61)] I0109 07:20:44.246600 1583 log_util.cc:368] Successfully rebuilt footer for segment: /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery/wal-000000001 (valid entries through byte offset 13802) I0109 07:20:44.246711 1583 tablet.cc:983] T 9e99b710634a44f4a948132a5a180600 Rewinding schema during bootstrap to Schema [ 0:key[string NOT NULL], 1:value[int32 NOT NULL] ] I0109 07:20:44.246983 1583 log.cc:351] Log is configured to *not* fsync() on all Append() calls I0109 07:20:44.249760 1583 tablet_bootstrap.cc:810] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e: Overwriting operations starting at: 2.50 up to: 2.50 with operation: 4.50 I0109 07:20:44.250186 1583 tablet_bootstrap.cc:381] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e: Bootstrap replayed 1/1 log segments. Stats: ops{read=62 overwritten=1 applied=61 ignored=0} inserts{seen=59 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I0109 07:20:44.250205 1583 tablet_bootstrap.cc:1017] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e: ReplayState: Previous OpId: 4.61, Committed OpId: 4.61, Pending Replicates: 0, Pending Commits: 0 I0109 07:20:44.250491 1583 tablet_bootstrap.cc:633] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e: Preparing to delete log recovery files and directory /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery I0109 07:20:44.250502 1583 tablet_bootstrap.cc:636] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e: Renaming log recovery dir from /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery to /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery-1483975244250501 I0109 07:20:44.250519 1583 tablet_bootstrap.cc:646] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e: Deleting all files from renamed log recovery directory /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery-1483975244250501 I0109 07:20:44.250609 1583 tablet_bootstrap.cc:649] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e: Completed deletion of old log recovery files and directory /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery-1483975244250501 I0109 07:20:44.250620 1583 tablet_bootstrap.cc:381] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e: Bootstrap complete. I0109 07:20:44.255466 1583 ts_tablet_manager.cc:730] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e: Time spent bootstrapping tablet: real 0.091s user 0.020s sys 0.028s I0109 07:20:44.255905 1583 raft_consensus.cc:287] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [term 4 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "710c236995db46699eb9055c19ac506e" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-4.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "1678276a36f849c28459de6b6788965a" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-5.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "591b1a1d383d4b659d7549653f4f136e" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-2.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "c5a3e756029444b2bf66ee0044f5bc35" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-1.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "7897578c54854ba0a5af32eb2aa04704" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-3.gce.cloudera.com" port: 7050 } } I0109 07:20:44.255939 1583 raft_consensus.cc:547] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [term 4 FOLLOWER]: Becoming Follower/Learner. State: Replica: 710c236995db46699eb9055c19ac506e, State: 1, Role: FOLLOWER I0109 07:20:44.255951 1583 consensus_queue.cc:175] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 61, Last appended: 4.61, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER I0109 07:20:44.256131 1583 ts_tablet_manager.cc:757] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s I0109 07:20:44.263142 1569 rpc_server.cc:164] RPC server started. Bound to: 0.0.0.0:7050 I0109 07:20:44.263206 1569 webserver.cc:126] Starting webserver on 0.0.0.0:8050 I0109 07:20:44.263212 1569 webserver.cc:137] Document root disabled I0109 07:20:44.263352 1569 webserver.cc:221] Webserver started. Bound to: http://0.0.0.0:8050/ I0109 07:20:44.263752 1569 tablet_server_main.cc:69] Tablet server successfully started. I0109 07:20:44.315841 1700 heartbeater.cc:292] Connected to a master server at aserbin-kudu-jepsen-master.gce.cloudera.com:7051 I0109 07:20:44.315901 1700 heartbeater.cc:359] Registering TS with master... I0109 07:20:44.315979 1700 heartbeater.cc:373] Master aserbin-kudu-jepsen-master.gce.cloudera.com:7051 requested a full tablet report, sending... I0109 07:20:44.441464 1624 raft_consensus.cc:410] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0109 07:20:44.441596 1624 raft_consensus.cc:2158] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [term 4 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 0.224s I0109 07:20:44.441691 1624 raft_consensus.cc:434] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "710c236995db46699eb9055c19ac506e" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-4.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "1678276a36f849c28459de6b6788965a" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-5.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "591b1a1d383d4b659d7549653f4f136e" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-2.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "c5a3e756029444b2bf66ee0044f5bc35" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-1.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "7897578c54854ba0a5af32eb2aa04704" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-3.gce.cloudera.com" port: 7050 } } I0109 07:20:44.642658 1624 leader_election.cc:215] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [CANDIDATE]: Term 5 pre-election: Requesting pre-vote from peer 1678276a36f849c28459de6b6788965a I0109 07:20:44.642935 1624 leader_election.cc:215] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [CANDIDATE]: Term 5 pre-election: Requesting pre-vote from peer 591b1a1d383d4b659d7549653f4f136e I0109 07:20:44.643074 1624 leader_election.cc:215] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [CANDIDATE]: Term 5 pre-election: Requesting pre-vote from peer c5a3e756029444b2bf66ee0044f5bc35 I0109 07:20:44.643167 1624 leader_election.cc:215] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [CANDIDATE]: Term 5 pre-election: Requesting pre-vote from peer 7897578c54854ba0a5af32eb2aa04704 W0109 07:20:44.644243 1581 leader_election.cc:271] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 7897578c54854ba0a5af32eb2aa04704: Network error: Client connection negotiation failed: client connection to 172.31.114.37:7050: connect: Connection refused (error 111) I0109 07:20:44.645638 1579 leader_election.cc:347] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [CANDIDATE]: Term 5 pre-election: Vote granted by peer 1678276a36f849c28459de6b6788965a I0109 07:20:44.704808 1581 leader_election.cc:347] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [CANDIDATE]: Term 5 pre-election: Vote granted by peer 591b1a1d383d4b659d7549653f4f136e I0109 07:20:44.704972 1581 leader_election.cc:242] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. I0109 07:20:44.705518 1706 raft_consensus.cc:2158] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [term 4 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 0.165s I0109 07:20:44.705554 1706 raft_consensus.cc:1968] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [term 4 FOLLOWER]: Leader pre-election won for term 5 I0109 07:20:44.705567 1706 raft_consensus.cc:410] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0109 07:20:44.705585 1706 raft_consensus.cc:2158] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [term 4 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 0.208s I0109 07:20:44.705597 1706 raft_consensus.cc:2202] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [term 4 FOLLOWER]: Advancing to term 5 I0109 07:20:44.706292 1706 raft_consensus.cc:434] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "710c236995db46699eb9055c19ac506e" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-4.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "1678276a36f849c28459de6b6788965a" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-5.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "591b1a1d383d4b659d7549653f4f136e" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-2.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "c5a3e756029444b2bf66ee0044f5bc35" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-1.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "7897578c54854ba0a5af32eb2aa04704" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-3.gce.cloudera.com" port: 7050 } } I0109 07:20:44.736084 1578 leader_election.cc:347] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [CANDIDATE]: Term 5 pre-election: Vote granted by peer c5a3e756029444b2bf66ee0044f5bc35 I0109 07:20:44.902941 1706 leader_election.cc:215] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [CANDIDATE]: Term 5 election: Requesting vote from peer 1678276a36f849c28459de6b6788965a I0109 07:20:44.902988 1706 leader_election.cc:215] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [CANDIDATE]: Term 5 election: Requesting vote from peer 591b1a1d383d4b659d7549653f4f136e I0109 07:20:44.903002 1706 leader_election.cc:215] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [CANDIDATE]: Term 5 election: Requesting vote from peer c5a3e756029444b2bf66ee0044f5bc35 I0109 07:20:44.903014 1706 leader_election.cc:215] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [CANDIDATE]: Term 5 election: Requesting vote from peer 7897578c54854ba0a5af32eb2aa04704 I0109 07:20:44.903049 1656 raft_consensus.cc:1674] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 591b1a1d383d4b659d7549653f4f136e in current term 5: Already voted for candidate 710c236995db46699eb9055c19ac506e in this term. W0109 07:20:44.903157 1656 rpcz_store.cc:234] Call kudu.consensus.ConsensusService.RequestConsensusVote from 172.31.113.2:56206 (request call id 1) took 197ms (client timeout 198). W0109 07:20:44.903198 1656 rpcz_store.cc:238] Trace: 0109 07:20:44.705887 (+ 0us) service_pool.cc:143] Inserting onto call queue 0109 07:20:44.705898 (+ 11us) service_pool.cc:202] Handling call 0109 07:20:44.903141 (+197243us) spinlock_profiling.cc:255] Waited 197 ms on lock 0x335f0f0. stack: 0000000001a20b21 00000000009d8b77 00000000007e48af 0000000000a8971f 0000000000a8a342 0000000001a2dc19 00007f973d4100a3 00007f973bca087c ffffffffffffffff 0109 07:20:44.903149 (+ 8us) inbound_call.cc:130] Queueing success response Metrics: {"spinlock_wait_cycles":492786944} I0109 07:20:44.903379 1655 raft_consensus.cc:1719] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c5a3e756029444b2bf66ee0044f5bc35 for term 5 because replica is already servicing an update from a current leader or another vote. W0109 07:20:44.903429 1655 rpcz_store.cc:234] Call kudu.consensus.ConsensusService.RequestConsensusVote from 172.31.112.255:38022 (request call id 1) took 165ms (client timeout 201). W0109 07:20:44.903453 1655 rpcz_store.cc:238] Trace: 0109 07:20:44.737743 (+ 0us) service_pool.cc:143] Inserting onto call queue 0109 07:20:44.737755 (+ 12us) service_pool.cc:202] Handling call 0109 07:20:44.903419 (+165664us) spinlock_profiling.cc:255] Waited 166 ms on lock 0x335f0f0. stack: 0000000001a20b21 00000000009d8c7a 00000000007e48af 0000000000a8971f 0000000000a8a342 0000000001a2dc19 00007f973d4100a3 00007f973bca087c ffffffffffffffff 0109 07:20:44.903424 (+ 5us) inbound_call.cc:130] Queueing success response Metrics: {"spinlock_wait_cycles":413974784} I0109 07:20:44.903808 1581 leader_election.cc:361] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [CANDIDATE]: Term 5 election: Vote denied by peer 591b1a1d383d4b659d7549653f4f136e. Message: Invalid argument: T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 710c236995db46699eb9055c19ac506e in current term 5: Already voted for candidate 591b1a1d383d4b659d7549653f4f136e in this term. I0109 07:20:44.903864 1655 raft_consensus.cc:1674] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 591b1a1d383d4b659d7549653f4f136e in current term 5: Already voted for candidate 710c236995db46699eb9055c19ac506e in this term. I0109 07:20:44.904718 1578 leader_election.cc:347] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [CANDIDATE]: Term 5 election: Vote granted by peer c5a3e756029444b2bf66ee0044f5bc35 I0109 07:20:44.905079 1579 leader_election.cc:347] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [CANDIDATE]: Term 5 election: Vote granted by peer 1678276a36f849c28459de6b6788965a I0109 07:20:44.905092 1579 leader_election.cc:242] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. I0109 07:20:44.905149 1706 raft_consensus.cc:2158] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [term 5 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 0.203s I0109 07:20:44.905164 1706 raft_consensus.cc:1968] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [term 5 FOLLOWER]: Leader election won for term 5 I0109 07:20:44.905190 1706 raft_consensus.cc:517] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [term 5 LEADER]: Becoming Leader. State: Replica: 710c236995db46699eb9055c19ac506e, State: 1, Role: LEADER I0109 07:20:44.905258 1706 consensus_queue.cc:157] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 61, Committed index: 61, Last appended: 4.61, Current term: 5, Majority size: 3, State: 1, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "710c236995db46699eb9055c19ac506e" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-4.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "1678276a36f849c28459de6b6788965a" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-5.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "591b1a1d383d4b659d7549653f4f136e" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-2.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "c5a3e756029444b2bf66ee0044f5bc35" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-1.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "7897578c54854ba0a5af32eb2aa04704" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-3.gce.cloudera.com" port: 7050 } } I0109 07:20:44.905365 1581 leader_election.cc:361] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [CANDIDATE]: Term 5 election: Vote denied by peer 7897578c54854ba0a5af32eb2aa04704. Message: Invalid argument: T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 710c236995db46699eb9055c19ac506e in current term 5: Already voted for candidate 591b1a1d383d4b659d7549653f4f136e in this term. I0109 07:20:45.103195 1700 heartbeater.cc:365] Master aserbin-kudu-jepsen-master.gce.cloudera.com:7051 was elected leader, sending a full tablet report... I0109 07:20:45.104074 1706 consensus_queue.cc:693] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [LEADER]: Connected to new peer: Peer: 1678276a36f849c28459de6b6788965a, Is new: false, Last received: 4.61, Next index: 62, Last known committed idx: 61, Last exchange result: ERROR, Needs tablet copy: false I0109 07:20:45.104133 1706 consensus_queue.cc:693] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [LEADER]: Connected to new peer: Peer: 591b1a1d383d4b659d7549653f4f136e, Is new: false, Last received: 4.61, Next index: 62, Last known committed idx: 61, Last exchange result: ERROR, Needs tablet copy: false I0109 07:20:45.104234 1706 consensus_queue.cc:693] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [LEADER]: Connected to new peer: Peer: c5a3e756029444b2bf66ee0044f5bc35, Is new: false, Last received: 1.49, Next index: 50, Last known committed idx: 49, Last exchange result: ERROR, Needs tablet copy: false I0109 07:20:45.105471 1706 consensus_queue.cc:693] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [LEADER]: Connected to new peer: Peer: 7897578c54854ba0a5af32eb2aa04704, Is new: false, Last received: 4.61, Next index: 62, Last known committed idx: 61, Last exchange result: ERROR, Needs tablet copy: false W0109 07:20:55.123915 1578 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer c5a3e756029444b2bf66ee0044f5bc35 (aserbin-kudu-jepsen-ts-1.gce.cloudera.com:7050): Couldn't send request to peer c5a3e756029444b2bf66ee0044f5bc35 for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.112.255:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 1 times. W0109 07:20:55.173842 1578 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer c5a3e756029444b2bf66ee0044f5bc35 (aserbin-kudu-jepsen-ts-1.gce.cloudera.com:7050): Couldn't send request to peer c5a3e756029444b2bf66ee0044f5bc35 for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.112.255:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 2 times. W0109 07:20:55.224019 1578 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer c5a3e756029444b2bf66ee0044f5bc35 (aserbin-kudu-jepsen-ts-1.gce.cloudera.com:7050): Couldn't send request to peer c5a3e756029444b2bf66ee0044f5bc35 for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.112.255:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 3 times. W0109 07:20:55.274117 1578 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer c5a3e756029444b2bf66ee0044f5bc35 (aserbin-kudu-jepsen-ts-1.gce.cloudera.com:7050): Couldn't send request to peer c5a3e756029444b2bf66ee0044f5bc35 for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.112.255:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 4 times. W0109 07:20:55.315382 1578 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer c5a3e756029444b2bf66ee0044f5bc35 (aserbin-kudu-jepsen-ts-1.gce.cloudera.com:7050): Couldn't send request to peer c5a3e756029444b2bf66ee0044f5bc35 for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.112.255:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 5 times. W0109 07:20:55.343528 1578 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer c5a3e756029444b2bf66ee0044f5bc35 (aserbin-kudu-jepsen-ts-1.gce.cloudera.com:7050): Couldn't send request to peer c5a3e756029444b2bf66ee0044f5bc35 for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.112.255:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 6 times. W0109 07:20:55.381765 1578 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer c5a3e756029444b2bf66ee0044f5bc35 (aserbin-kudu-jepsen-ts-1.gce.cloudera.com:7050): Couldn't send request to peer c5a3e756029444b2bf66ee0044f5bc35 for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.112.255:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 7 times. W0109 07:20:55.409801 1578 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer c5a3e756029444b2bf66ee0044f5bc35 (aserbin-kudu-jepsen-ts-1.gce.cloudera.com:7050): Couldn't send request to peer c5a3e756029444b2bf66ee0044f5bc35 for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.112.255:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 8 times. W0109 07:20:55.438046 1578 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer c5a3e756029444b2bf66ee0044f5bc35 (aserbin-kudu-jepsen-ts-1.gce.cloudera.com:7050): Couldn't send request to peer c5a3e756029444b2bf66ee0044f5bc35 for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.112.255:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 9 times. W0109 07:20:55.474146 1578 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer c5a3e756029444b2bf66ee0044f5bc35 (aserbin-kudu-jepsen-ts-1.gce.cloudera.com:7050): Couldn't send request to peer c5a3e756029444b2bf66ee0044f5bc35 for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.112.255:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 10 times. W0109 07:20:55.518345 1578 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer c5a3e756029444b2bf66ee0044f5bc35 (aserbin-kudu-jepsen-ts-1.gce.cloudera.com:7050): Couldn't send request to peer c5a3e756029444b2bf66ee0044f5bc35 for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.112.255:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 11 times. W0109 07:20:55.555431 1578 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer c5a3e756029444b2bf66ee0044f5bc35 (aserbin-kudu-jepsen-ts-1.gce.cloudera.com:7050): Couldn't send request to peer c5a3e756029444b2bf66ee0044f5bc35 for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.112.255:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 12 times. W0109 07:20:55.587890 1578 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer c5a3e756029444b2bf66ee0044f5bc35 (aserbin-kudu-jepsen-ts-1.gce.cloudera.com:7050): Couldn't send request to peer c5a3e756029444b2bf66ee0044f5bc35 for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.112.255:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 13 times. W0109 07:20:55.629931 1578 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer c5a3e756029444b2bf66ee0044f5bc35 (aserbin-kudu-jepsen-ts-1.gce.cloudera.com:7050): Couldn't send request to peer c5a3e756029444b2bf66ee0044f5bc35 for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.112.255:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 14 times. W0109 07:20:55.664019 1578 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer c5a3e756029444b2bf66ee0044f5bc35 (aserbin-kudu-jepsen-ts-1.gce.cloudera.com:7050): Couldn't send request to peer c5a3e756029444b2bf66ee0044f5bc35 for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.112.255:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 15 times. W0109 07:20:55.692311 1578 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer c5a3e756029444b2bf66ee0044f5bc35 (aserbin-kudu-jepsen-ts-1.gce.cloudera.com:7050): Couldn't send request to peer c5a3e756029444b2bf66ee0044f5bc35 for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.112.255:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 16 times. W0109 07:20:55.721534 1578 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer c5a3e756029444b2bf66ee0044f5bc35 (aserbin-kudu-jepsen-ts-1.gce.cloudera.com:7050): Couldn't send request to peer c5a3e756029444b2bf66ee0044f5bc35 for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.112.255:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 17 times. W0109 07:20:55.761642 1578 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer c5a3e756029444b2bf66ee0044f5bc35 (aserbin-kudu-jepsen-ts-1.gce.cloudera.com:7050): Couldn't send request to peer c5a3e756029444b2bf66ee0044f5bc35 for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.112.255:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 18 times. W0109 07:20:55.797679 1578 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer c5a3e756029444b2bf66ee0044f5bc35 (aserbin-kudu-jepsen-ts-1.gce.cloudera.com:7050): Couldn't send request to peer c5a3e756029444b2bf66ee0044f5bc35 for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.112.255:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 19 times. W0109 07:20:55.827836 1578 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer c5a3e756029444b2bf66ee0044f5bc35 (aserbin-kudu-jepsen-ts-1.gce.cloudera.com:7050): Couldn't send request to peer c5a3e756029444b2bf66ee0044f5bc35 for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.112.255:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 20 times. W0109 07:20:55.856042 1578 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer c5a3e756029444b2bf66ee0044f5bc35 (aserbin-kudu-jepsen-ts-1.gce.cloudera.com:7050): Couldn't send request to peer c5a3e756029444b2bf66ee0044f5bc35 for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.112.255:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 21 times. W0109 07:20:55.888273 1578 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer c5a3e756029444b2bf66ee0044f5bc35 (aserbin-kudu-jepsen-ts-1.gce.cloudera.com:7050): Couldn't send request to peer c5a3e756029444b2bf66ee0044f5bc35 for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.112.255:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 22 times. W0109 07:20:55.913357 1578 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer c5a3e756029444b2bf66ee0044f5bc35 (aserbin-kudu-jepsen-ts-1.gce.cloudera.com:7050): Couldn't send request to peer c5a3e756029444b2bf66ee0044f5bc35 for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.112.255:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 23 times. W0109 07:20:55.940596 1578 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer c5a3e756029444b2bf66ee0044f5bc35 (aserbin-kudu-jepsen-ts-1.gce.cloudera.com:7050): Couldn't send request to peer c5a3e756029444b2bf66ee0044f5bc35 for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.112.255:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 24 times. W0109 07:20:55.970693 1578 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer c5a3e756029444b2bf66ee0044f5bc35 (aserbin-kudu-jepsen-ts-1.gce.cloudera.com:7050): Couldn't send request to peer c5a3e756029444b2bf66ee0044f5bc35 for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.112.255:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 25 times. W0109 07:20:56.007971 1578 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer c5a3e756029444b2bf66ee0044f5bc35 (aserbin-kudu-jepsen-ts-1.gce.cloudera.com:7050): Couldn't send request to peer c5a3e756029444b2bf66ee0044f5bc35 for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.112.255:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 26 times. W0109 07:20:56.050122 1578 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer c5a3e756029444b2bf66ee0044f5bc35 (aserbin-kudu-jepsen-ts-1.gce.cloudera.com:7050): Couldn't send request to peer c5a3e756029444b2bf66ee0044f5bc35 for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.112.255:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 27 times. W0109 07:20:56.083238 1578 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer c5a3e756029444b2bf66ee0044f5bc35 (aserbin-kudu-jepsen-ts-1.gce.cloudera.com:7050): Couldn't send request to peer c5a3e756029444b2bf66ee0044f5bc35 for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.112.255:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 28 times. W0109 07:20:56.120687 1578 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer c5a3e756029444b2bf66ee0044f5bc35 (aserbin-kudu-jepsen-ts-1.gce.cloudera.com:7050): Couldn't send request to peer c5a3e756029444b2bf66ee0044f5bc35 for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.112.255:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 29 times. W0109 07:20:56.168602 1578 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer c5a3e756029444b2bf66ee0044f5bc35 (aserbin-kudu-jepsen-ts-1.gce.cloudera.com:7050): Couldn't send request to peer c5a3e756029444b2bf66ee0044f5bc35 for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.112.255:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 30 times. W0109 07:20:56.211797 1578 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer c5a3e756029444b2bf66ee0044f5bc35 (aserbin-kudu-jepsen-ts-1.gce.cloudera.com:7050): Couldn't send request to peer c5a3e756029444b2bf66ee0044f5bc35 for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.112.255:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 31 times. W0109 07:20:56.241032 1578 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer c5a3e756029444b2bf66ee0044f5bc35 (aserbin-kudu-jepsen-ts-1.gce.cloudera.com:7050): Couldn't send request to peer c5a3e756029444b2bf66ee0044f5bc35 for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.112.255:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 32 times. W0109 07:20:56.281143 1578 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer c5a3e756029444b2bf66ee0044f5bc35 (aserbin-kudu-jepsen-ts-1.gce.cloudera.com:7050): Couldn't send request to peer c5a3e756029444b2bf66ee0044f5bc35 for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.112.255:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 33 times. W0109 07:20:56.314250 1578 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer c5a3e756029444b2bf66ee0044f5bc35 (aserbin-kudu-jepsen-ts-1.gce.cloudera.com:7050): Couldn't send request to peer c5a3e756029444b2bf66ee0044f5bc35 for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.112.255:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 34 times. W0109 07:20:56.344519 1578 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer c5a3e756029444b2bf66ee0044f5bc35 (aserbin-kudu-jepsen-ts-1.gce.cloudera.com:7050): Couldn't send request to peer c5a3e756029444b2bf66ee0044f5bc35 for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.112.255:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 35 times. W0109 07:20:56.370622 1578 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer c5a3e756029444b2bf66ee0044f5bc35 (aserbin-kudu-jepsen-ts-1.gce.cloudera.com:7050): Couldn't send request to peer c5a3e756029444b2bf66ee0044f5bc35 for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.112.255:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 36 times. W0109 07:20:56.408808 1578 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer c5a3e756029444b2bf66ee0044f5bc35 (aserbin-kudu-jepsen-ts-1.gce.cloudera.com:7050): Couldn't send request to peer c5a3e756029444b2bf66ee0044f5bc35 for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.112.255:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 37 times. W0109 07:20:56.457021 1578 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer c5a3e756029444b2bf66ee0044f5bc35 (aserbin-kudu-jepsen-ts-1.gce.cloudera.com:7050): Couldn't send request to peer c5a3e756029444b2bf66ee0044f5bc35 for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.112.255:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 38 times. W0109 07:20:56.495146 1578 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer c5a3e756029444b2bf66ee0044f5bc35 (aserbin-kudu-jepsen-ts-1.gce.cloudera.com:7050): Couldn't send request to peer c5a3e756029444b2bf66ee0044f5bc35 for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.112.255:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 39 times. W0109 07:20:56.537294 1578 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer c5a3e756029444b2bf66ee0044f5bc35 (aserbin-kudu-jepsen-ts-1.gce.cloudera.com:7050): Couldn't send request to peer c5a3e756029444b2bf66ee0044f5bc35 for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.112.255:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 40 times. W0109 07:20:56.562520 1578 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer c5a3e756029444b2bf66ee0044f5bc35 (aserbin-kudu-jepsen-ts-1.gce.cloudera.com:7050): Couldn't send request to peer c5a3e756029444b2bf66ee0044f5bc35 for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.112.255:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 41 times. W0109 07:20:56.611856 1578 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer c5a3e756029444b2bf66ee0044f5bc35 (aserbin-kudu-jepsen-ts-1.gce.cloudera.com:7050): Couldn't send request to peer c5a3e756029444b2bf66ee0044f5bc35 for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.112.255:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 42 times. W0109 07:20:56.652201 1578 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer c5a3e756029444b2bf66ee0044f5bc35 (aserbin-kudu-jepsen-ts-1.gce.cloudera.com:7050): Couldn't send request to peer c5a3e756029444b2bf66ee0044f5bc35 for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.112.255:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 43 times. W0109 07:20:56.685412 1578 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer c5a3e756029444b2bf66ee0044f5bc35 (aserbin-kudu-jepsen-ts-1.gce.cloudera.com:7050): Couldn't send request to peer c5a3e756029444b2bf66ee0044f5bc35 for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.112.255:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 44 times. W0109 07:20:56.730466 1578 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer c5a3e756029444b2bf66ee0044f5bc35 (aserbin-kudu-jepsen-ts-1.gce.cloudera.com:7050): Couldn't send request to peer c5a3e756029444b2bf66ee0044f5bc35 for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.112.255:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 45 times. W0109 07:20:56.769515 1578 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer c5a3e756029444b2bf66ee0044f5bc35 (aserbin-kudu-jepsen-ts-1.gce.cloudera.com:7050): Couldn't send request to peer c5a3e756029444b2bf66ee0044f5bc35 for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.112.255:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 46 times. W0109 07:20:56.817730 1578 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer c5a3e756029444b2bf66ee0044f5bc35 (aserbin-kudu-jepsen-ts-1.gce.cloudera.com:7050): Couldn't send request to peer c5a3e756029444b2bf66ee0044f5bc35 for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.112.255:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 47 times. W0109 07:20:56.864802 1578 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer c5a3e756029444b2bf66ee0044f5bc35 (aserbin-kudu-jepsen-ts-1.gce.cloudera.com:7050): Couldn't send request to peer c5a3e756029444b2bf66ee0044f5bc35 for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.112.255:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 48 times. W0109 07:20:56.909929 1578 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer c5a3e756029444b2bf66ee0044f5bc35 (aserbin-kudu-jepsen-ts-1.gce.cloudera.com:7050): Couldn't send request to peer c5a3e756029444b2bf66ee0044f5bc35 for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.112.255:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 49 times. W0109 07:20:56.948226 1578 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer c5a3e756029444b2bf66ee0044f5bc35 (aserbin-kudu-jepsen-ts-1.gce.cloudera.com:7050): Couldn't send request to peer c5a3e756029444b2bf66ee0044f5bc35 for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.112.255:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 50 times. W0109 07:20:56.988519 1578 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer c5a3e756029444b2bf66ee0044f5bc35 (aserbin-kudu-jepsen-ts-1.gce.cloudera.com:7050): Couldn't send request to peer c5a3e756029444b2bf66ee0044f5bc35 for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.112.255:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 51 times. W0109 07:20:57.026520 1578 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer c5a3e756029444b2bf66ee0044f5bc35 (aserbin-kudu-jepsen-ts-1.gce.cloudera.com:7050): Couldn't send request to peer c5a3e756029444b2bf66ee0044f5bc35 for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.112.255:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 52 times. W0109 07:20:57.066681 1578 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer c5a3e756029444b2bf66ee0044f5bc35 (aserbin-kudu-jepsen-ts-1.gce.cloudera.com:7050): Couldn't send request to peer c5a3e756029444b2bf66ee0044f5bc35 for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.112.255:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 53 times. W0109 07:20:57.101835 1578 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer c5a3e756029444b2bf66ee0044f5bc35 (aserbin-kudu-jepsen-ts-1.gce.cloudera.com:7050): Couldn't send request to peer c5a3e756029444b2bf66ee0044f5bc35 for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.112.255:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 54 times. W0109 07:20:57.143026 1578 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer c5a3e756029444b2bf66ee0044f5bc35 (aserbin-kudu-jepsen-ts-1.gce.cloudera.com:7050): Couldn't send request to peer c5a3e756029444b2bf66ee0044f5bc35 for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.112.255:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 55 times. W0109 07:20:57.190522 1578 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer c5a3e756029444b2bf66ee0044f5bc35 (aserbin-kudu-jepsen-ts-1.gce.cloudera.com:7050): Couldn't send request to peer c5a3e756029444b2bf66ee0044f5bc35 for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.112.255:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 56 times. W0109 07:20:57.219343 1578 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer c5a3e756029444b2bf66ee0044f5bc35 (aserbin-kudu-jepsen-ts-1.gce.cloudera.com:7050): Couldn't send request to peer c5a3e756029444b2bf66ee0044f5bc35 for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.112.255:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 57 times. W0109 07:20:57.258652 1578 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer c5a3e756029444b2bf66ee0044f5bc35 (aserbin-kudu-jepsen-ts-1.gce.cloudera.com:7050): Couldn't send request to peer c5a3e756029444b2bf66ee0044f5bc35 for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.112.255:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 58 times. W0109 07:20:57.307775 1578 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer c5a3e756029444b2bf66ee0044f5bc35 (aserbin-kudu-jepsen-ts-1.gce.cloudera.com:7050): Couldn't send request to peer c5a3e756029444b2bf66ee0044f5bc35 for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.112.255:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 59 times. W0109 07:20:57.335932 1578 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer c5a3e756029444b2bf66ee0044f5bc35 (aserbin-kudu-jepsen-ts-1.gce.cloudera.com:7050): Couldn't send request to peer c5a3e756029444b2bf66ee0044f5bc35 for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.112.255:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 60 times. W0109 07:20:57.381181 1578 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer c5a3e756029444b2bf66ee0044f5bc35 (aserbin-kudu-jepsen-ts-1.gce.cloudera.com:7050): Couldn't send request to peer c5a3e756029444b2bf66ee0044f5bc35 for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.112.255:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 61 times. W0109 07:20:57.424281 1578 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer c5a3e756029444b2bf66ee0044f5bc35 (aserbin-kudu-jepsen-ts-1.gce.cloudera.com:7050): Couldn't send request to peer c5a3e756029444b2bf66ee0044f5bc35 for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.112.255:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 62 times. W0109 07:20:57.470451 1578 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer c5a3e756029444b2bf66ee0044f5bc35 (aserbin-kudu-jepsen-ts-1.gce.cloudera.com:7050): Couldn't send request to peer c5a3e756029444b2bf66ee0044f5bc35 for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.112.255:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 63 times. W0109 07:20:57.507755 1578 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer c5a3e756029444b2bf66ee0044f5bc35 (aserbin-kudu-jepsen-ts-1.gce.cloudera.com:7050): Couldn't send request to peer c5a3e756029444b2bf66ee0044f5bc35 for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.112.255:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 64 times. W0109 07:20:57.553995 1578 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer c5a3e756029444b2bf66ee0044f5bc35 (aserbin-kudu-jepsen-ts-1.gce.cloudera.com:7050): Couldn't send request to peer c5a3e756029444b2bf66ee0044f5bc35 for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.112.255:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 65 times. W0109 07:20:57.590159 1578 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer c5a3e756029444b2bf66ee0044f5bc35 (aserbin-kudu-jepsen-ts-1.gce.cloudera.com:7050): Couldn't send request to peer c5a3e756029444b2bf66ee0044f5bc35 for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.112.255:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 66 times. W0109 07:20:57.628379 1578 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer c5a3e756029444b2bf66ee0044f5bc35 (aserbin-kudu-jepsen-ts-1.gce.cloudera.com:7050): Couldn't send request to peer c5a3e756029444b2bf66ee0044f5bc35 for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.112.255:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 67 times. W0109 07:20:57.665575 1578 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer c5a3e756029444b2bf66ee0044f5bc35 (aserbin-kudu-jepsen-ts-1.gce.cloudera.com:7050): Couldn't send request to peer c5a3e756029444b2bf66ee0044f5bc35 for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.112.255:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 68 times. W0109 07:20:57.711693 1578 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer c5a3e756029444b2bf66ee0044f5bc35 (aserbin-kudu-jepsen-ts-1.gce.cloudera.com:7050): Couldn't send request to peer c5a3e756029444b2bf66ee0044f5bc35 for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.112.255:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 69 times. W0109 07:20:57.750790 1578 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer c5a3e756029444b2bf66ee0044f5bc35 (aserbin-kudu-jepsen-ts-1.gce.cloudera.com:7050): Couldn't send request to peer c5a3e756029444b2bf66ee0044f5bc35 for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.112.255:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 70 times. W0109 07:20:57.798939 1578 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer c5a3e756029444b2bf66ee0044f5bc35 (aserbin-kudu-jepsen-ts-1.gce.cloudera.com:7050): Couldn't send request to peer c5a3e756029444b2bf66ee0044f5bc35 for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.112.255:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 71 times. W0109 07:20:57.826262 1578 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer c5a3e756029444b2bf66ee0044f5bc35 (aserbin-kudu-jepsen-ts-1.gce.cloudera.com:7050): Couldn't send request to peer c5a3e756029444b2bf66ee0044f5bc35 for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.112.255:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 72 times. W0109 07:20:57.864320 1578 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer c5a3e756029444b2bf66ee0044f5bc35 (aserbin-kudu-jepsen-ts-1.gce.cloudera.com:7050): Couldn't send request to peer c5a3e756029444b2bf66ee0044f5bc35 for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.112.255:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 73 times. W0109 07:20:57.900483 1578 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer c5a3e756029444b2bf66ee0044f5bc35 (aserbin-kudu-jepsen-ts-1.gce.cloudera.com:7050): Couldn't send request to peer c5a3e756029444b2bf66ee0044f5bc35 for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.112.255:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 74 times. W0109 07:20:57.946589 1578 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer c5a3e756029444b2bf66ee0044f5bc35 (aserbin-kudu-jepsen-ts-1.gce.cloudera.com:7050): Couldn't send request to peer c5a3e756029444b2bf66ee0044f5bc35 for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.112.255:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 75 times. W0109 07:20:57.993775 1578 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer c5a3e756029444b2bf66ee0044f5bc35 (aserbin-kudu-jepsen-ts-1.gce.cloudera.com:7050): Couldn't send request to peer c5a3e756029444b2bf66ee0044f5bc35 for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.112.255:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 76 times. W0109 07:20:58.029894 1578 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer c5a3e756029444b2bf66ee0044f5bc35 (aserbin-kudu-jepsen-ts-1.gce.cloudera.com:7050): Couldn't send request to peer c5a3e756029444b2bf66ee0044f5bc35 for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.112.255:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 77 times. W0109 07:20:58.078081 1578 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer c5a3e756029444b2bf66ee0044f5bc35 (aserbin-kudu-jepsen-ts-1.gce.cloudera.com:7050): Couldn't send request to peer c5a3e756029444b2bf66ee0044f5bc35 for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.112.255:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 78 times. W0109 07:20:58.107224 1578 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer c5a3e756029444b2bf66ee0044f5bc35 (aserbin-kudu-jepsen-ts-1.gce.cloudera.com:7050): Couldn't send request to peer c5a3e756029444b2bf66ee0044f5bc35 for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.112.255:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 79 times. W0109 07:20:58.139397 1578 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer c5a3e756029444b2bf66ee0044f5bc35 (aserbin-kudu-jepsen-ts-1.gce.cloudera.com:7050): Couldn't send request to peer c5a3e756029444b2bf66ee0044f5bc35 for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.112.255:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 80 times. W0109 07:20:58.172549 1578 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer c5a3e756029444b2bf66ee0044f5bc35 (aserbin-kudu-jepsen-ts-1.gce.cloudera.com:7050): Couldn't send request to peer c5a3e756029444b2bf66ee0044f5bc35 for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.112.255:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 81 times. W0109 07:20:58.215615 1578 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer c5a3e756029444b2bf66ee0044f5bc35 (aserbin-kudu-jepsen-ts-1.gce.cloudera.com:7050): Couldn't send request to peer c5a3e756029444b2bf66ee0044f5bc35 for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.112.255:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 82 times. W0109 07:20:58.254740 1578 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer c5a3e756029444b2bf66ee0044f5bc35 (aserbin-kudu-jepsen-ts-1.gce.cloudera.com:7050): Couldn't send request to peer c5a3e756029444b2bf66ee0044f5bc35 for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.112.255:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 83 times. W0109 07:20:58.290856 1578 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer c5a3e756029444b2bf66ee0044f5bc35 (aserbin-kudu-jepsen-ts-1.gce.cloudera.com:7050): Couldn't send request to peer c5a3e756029444b2bf66ee0044f5bc35 for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.112.255:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 84 times. W0109 07:20:58.339085 1578 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer c5a3e756029444b2bf66ee0044f5bc35 (aserbin-kudu-jepsen-ts-1.gce.cloudera.com:7050): Couldn't send request to peer c5a3e756029444b2bf66ee0044f5bc35 for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.112.255:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 85 times. W0109 07:20:58.372233 1578 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer c5a3e756029444b2bf66ee0044f5bc35 (aserbin-kudu-jepsen-ts-1.gce.cloudera.com:7050): Couldn't send request to peer c5a3e756029444b2bf66ee0044f5bc35 for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.112.255:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 86 times. W0109 07:20:58.409907 1578 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer c5a3e756029444b2bf66ee0044f5bc35 (aserbin-kudu-jepsen-ts-1.gce.cloudera.com:7050): Couldn't send request to peer c5a3e756029444b2bf66ee0044f5bc35 for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.112.255:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 87 times. W0109 07:20:58.439574 1578 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer c5a3e756029444b2bf66ee0044f5bc35 (aserbin-kudu-jepsen-ts-1.gce.cloudera.com:7050): Couldn't send request to peer c5a3e756029444b2bf66ee0044f5bc35 for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.112.255:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 88 times. W0109 07:20:58.464619 1578 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer c5a3e756029444b2bf66ee0044f5bc35 (aserbin-kudu-jepsen-ts-1.gce.cloudera.com:7050): Couldn't send request to peer c5a3e756029444b2bf66ee0044f5bc35 for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.112.255:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 89 times. W0109 07:20:58.494839 1578 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer c5a3e756029444b2bf66ee0044f5bc35 (aserbin-kudu-jepsen-ts-1.gce.cloudera.com:7050): Couldn't send request to peer c5a3e756029444b2bf66ee0044f5bc35 for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.112.255:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 90 times. W0109 07:20:58.526955 1578 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer c5a3e756029444b2bf66ee0044f5bc35 (aserbin-kudu-jepsen-ts-1.gce.cloudera.com:7050): Couldn't send request to peer c5a3e756029444b2bf66ee0044f5bc35 for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.112.255:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 91 times. W0109 07:20:58.554181 1578 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer c5a3e756029444b2bf66ee0044f5bc35 (aserbin-kudu-jepsen-ts-1.gce.cloudera.com:7050): Couldn't send request to peer c5a3e756029444b2bf66ee0044f5bc35 for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.112.255:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 92 times. W0109 07:20:58.593343 1578 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer c5a3e756029444b2bf66ee0044f5bc35 (aserbin-kudu-jepsen-ts-1.gce.cloudera.com:7050): Couldn't send request to peer c5a3e756029444b2bf66ee0044f5bc35 for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.112.255:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 93 times. W0109 07:20:58.628515 1578 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer c5a3e756029444b2bf66ee0044f5bc35 (aserbin-kudu-jepsen-ts-1.gce.cloudera.com:7050): Couldn't send request to peer c5a3e756029444b2bf66ee0044f5bc35 for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.112.255:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 94 times. W0109 07:20:58.654667 1578 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer c5a3e756029444b2bf66ee0044f5bc35 (aserbin-kudu-jepsen-ts-1.gce.cloudera.com:7050): Couldn't send request to peer c5a3e756029444b2bf66ee0044f5bc35 for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.112.255:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 95 times. W0109 07:20:58.686719 1578 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer c5a3e756029444b2bf66ee0044f5bc35 (aserbin-kudu-jepsen-ts-1.gce.cloudera.com:7050): Couldn't send request to peer c5a3e756029444b2bf66ee0044f5bc35 for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.112.255:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 96 times. W0109 07:20:58.724869 1578 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer c5a3e756029444b2bf66ee0044f5bc35 (aserbin-kudu-jepsen-ts-1.gce.cloudera.com:7050): Couldn't send request to peer c5a3e756029444b2bf66ee0044f5bc35 for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.112.255:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 97 times. W0109 07:20:58.757295 1578 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer c5a3e756029444b2bf66ee0044f5bc35 (aserbin-kudu-jepsen-ts-1.gce.cloudera.com:7050): Couldn't send request to peer c5a3e756029444b2bf66ee0044f5bc35 for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.112.255:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 98 times. W0109 07:20:58.802379 1578 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer c5a3e756029444b2bf66ee0044f5bc35 (aserbin-kudu-jepsen-ts-1.gce.cloudera.com:7050): Couldn't send request to peer c5a3e756029444b2bf66ee0044f5bc35 for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.112.255:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 99 times. W0109 07:20:58.838564 1578 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer c5a3e756029444b2bf66ee0044f5bc35 (aserbin-kudu-jepsen-ts-1.gce.cloudera.com:7050): Couldn't send request to peer c5a3e756029444b2bf66ee0044f5bc35 for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.112.255:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 100 times. W0109 07:20:58.868625 1578 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer c5a3e756029444b2bf66ee0044f5bc35 (aserbin-kudu-jepsen-ts-1.gce.cloudera.com:7050): Couldn't send request to peer c5a3e756029444b2bf66ee0044f5bc35 for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.112.255:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 101 times. W0109 07:20:58.917807 1578 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer c5a3e756029444b2bf66ee0044f5bc35 (aserbin-kudu-jepsen-ts-1.gce.cloudera.com:7050): Couldn't send request to peer c5a3e756029444b2bf66ee0044f5bc35 for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.112.255:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 102 times. W0109 07:20:58.952986 1578 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer c5a3e756029444b2bf66ee0044f5bc35 (aserbin-kudu-jepsen-ts-1.gce.cloudera.com:7050): Couldn't send request to peer c5a3e756029444b2bf66ee0044f5bc35 for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.112.255:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 103 times. W0109 07:20:58.997201 1578 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer c5a3e756029444b2bf66ee0044f5bc35 (aserbin-kudu-jepsen-ts-1.gce.cloudera.com:7050): Couldn't send request to peer c5a3e756029444b2bf66ee0044f5bc35 for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.112.255:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 104 times. W0109 07:20:59.035331 1578 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer c5a3e756029444b2bf66ee0044f5bc35 (aserbin-kudu-jepsen-ts-1.gce.cloudera.com:7050): Couldn't send request to peer c5a3e756029444b2bf66ee0044f5bc35 for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.112.255:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 105 times. W0109 07:20:59.067574 1578 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer c5a3e756029444b2bf66ee0044f5bc35 (aserbin-kudu-jepsen-ts-1.gce.cloudera.com:7050): Couldn't send request to peer c5a3e756029444b2bf66ee0044f5bc35 for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.112.255:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 106 times. W0109 07:20:59.102695 1578 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer c5a3e756029444b2bf66ee0044f5bc35 (aserbin-kudu-jepsen-ts-1.gce.cloudera.com:7050): Couldn't send request to peer c5a3e756029444b2bf66ee0044f5bc35 for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.112.255:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 107 times. W0109 07:20:59.151010 1578 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer c5a3e756029444b2bf66ee0044f5bc35 (aserbin-kudu-jepsen-ts-1.gce.cloudera.com:7050): Couldn't send request to peer c5a3e756029444b2bf66ee0044f5bc35 for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.112.255:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 108 times. W0109 07:20:59.190093 1578 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer c5a3e756029444b2bf66ee0044f5bc35 (aserbin-kudu-jepsen-ts-1.gce.cloudera.com:7050): Couldn't send request to peer c5a3e756029444b2bf66ee0044f5bc35 for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.112.255:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 109 times. W0109 07:20:59.225291 1578 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer c5a3e756029444b2bf66ee0044f5bc35 (aserbin-kudu-jepsen-ts-1.gce.cloudera.com:7050): Couldn't send request to peer c5a3e756029444b2bf66ee0044f5bc35 for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.112.255:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 110 times. W0109 07:20:59.269536 1578 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer c5a3e756029444b2bf66ee0044f5bc35 (aserbin-kudu-jepsen-ts-1.gce.cloudera.com:7050): Couldn't send request to peer c5a3e756029444b2bf66ee0044f5bc35 for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.112.255:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 111 times. W0109 07:20:59.309662 1578 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer c5a3e756029444b2bf66ee0044f5bc35 (aserbin-kudu-jepsen-ts-1.gce.cloudera.com:7050): Couldn't send request to peer c5a3e756029444b2bf66ee0044f5bc35 for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.112.255:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 112 times. W0109 07:20:59.342890 1578 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer c5a3e756029444b2bf66ee0044f5bc35 (aserbin-kudu-jepsen-ts-1.gce.cloudera.com:7050): Couldn't send request to peer c5a3e756029444b2bf66ee0044f5bc35 for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.112.255:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 113 times. W0109 07:20:59.385998 1578 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer c5a3e756029444b2bf66ee0044f5bc35 (aserbin-kudu-jepsen-ts-1.gce.cloudera.com:7050): Couldn't send request to peer c5a3e756029444b2bf66ee0044f5bc35 for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.112.255:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 114 times. W0109 07:20:59.435158 1578 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer c5a3e756029444b2bf66ee0044f5bc35 (aserbin-kudu-jepsen-ts-1.gce.cloudera.com:7050): Couldn't send request to peer c5a3e756029444b2bf66ee0044f5bc35 for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.112.255:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 115 times. W0109 07:20:59.474359 1578 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer c5a3e756029444b2bf66ee0044f5bc35 (aserbin-kudu-jepsen-ts-1.gce.cloudera.com:7050): Couldn't send request to peer c5a3e756029444b2bf66ee0044f5bc35 for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.112.255:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 116 times. W0109 07:20:59.502454 1578 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer c5a3e756029444b2bf66ee0044f5bc35 (aserbin-kudu-jepsen-ts-1.gce.cloudera.com:7050): Couldn't send request to peer c5a3e756029444b2bf66ee0044f5bc35 for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.112.255:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 117 times. W0109 07:20:59.540063 1578 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer c5a3e756029444b2bf66ee0044f5bc35 (aserbin-kudu-jepsen-ts-1.gce.cloudera.com:7050): Couldn't send request to peer c5a3e756029444b2bf66ee0044f5bc35 for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.112.255:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 118 times. W0109 07:20:59.581995 1578 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer c5a3e756029444b2bf66ee0044f5bc35 (aserbin-kudu-jepsen-ts-1.gce.cloudera.com:7050): Couldn't send request to peer c5a3e756029444b2bf66ee0044f5bc35 for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.112.255:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 119 times. W0109 07:20:59.608997 1578 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer c5a3e756029444b2bf66ee0044f5bc35 (aserbin-kudu-jepsen-ts-1.gce.cloudera.com:7050): Couldn't send request to peer c5a3e756029444b2bf66ee0044f5bc35 for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.112.255:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 120 times. W0109 07:20:59.646338 1578 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer c5a3e756029444b2bf66ee0044f5bc35 (aserbin-kudu-jepsen-ts-1.gce.cloudera.com:7050): Couldn't send request to peer c5a3e756029444b2bf66ee0044f5bc35 for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.112.255:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 121 times. W0109 07:20:59.683441 1578 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer c5a3e756029444b2bf66ee0044f5bc35 (aserbin-kudu-jepsen-ts-1.gce.cloudera.com:7050): Couldn't send request to peer c5a3e756029444b2bf66ee0044f5bc35 for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.112.255:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 122 times. W0109 07:20:59.716522 1578 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer c5a3e756029444b2bf66ee0044f5bc35 (aserbin-kudu-jepsen-ts-1.gce.cloudera.com:7050): Couldn't send request to peer c5a3e756029444b2bf66ee0044f5bc35 for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.112.255:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 123 times. W0109 07:20:59.741746 1578 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer c5a3e756029444b2bf66ee0044f5bc35 (aserbin-kudu-jepsen-ts-1.gce.cloudera.com:7050): Couldn't send request to peer c5a3e756029444b2bf66ee0044f5bc35 for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.112.255:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 124 times. W0109 07:20:59.785995 1578 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer c5a3e756029444b2bf66ee0044f5bc35 (aserbin-kudu-jepsen-ts-1.gce.cloudera.com:7050): Couldn't send request to peer c5a3e756029444b2bf66ee0044f5bc35 for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.112.255:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 125 times. W0109 07:20:59.816115 1578 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer c5a3e756029444b2bf66ee0044f5bc35 (aserbin-kudu-jepsen-ts-1.gce.cloudera.com:7050): Couldn't send request to peer c5a3e756029444b2bf66ee0044f5bc35 for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.112.255:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 126 times. W0109 07:20:59.854326 1578 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer c5a3e756029444b2bf66ee0044f5bc35 (aserbin-kudu-jepsen-ts-1.gce.cloudera.com:7050): Couldn't send request to peer c5a3e756029444b2bf66ee0044f5bc35 for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.112.255:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 127 times. W0109 07:20:59.897462 1578 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer c5a3e756029444b2bf66ee0044f5bc35 (aserbin-kudu-jepsen-ts-1.gce.cloudera.com:7050): Couldn't send request to peer c5a3e756029444b2bf66ee0044f5bc35 for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.112.255:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 128 times. W0109 07:20:59.935544 1578 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer c5a3e756029444b2bf66ee0044f5bc35 (aserbin-kudu-jepsen-ts-1.gce.cloudera.com:7050): Couldn't send request to peer c5a3e756029444b2bf66ee0044f5bc35 for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.112.255:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 129 times. W0109 07:20:59.967828 1578 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer c5a3e756029444b2bf66ee0044f5bc35 (aserbin-kudu-jepsen-ts-1.gce.cloudera.com:7050): Couldn't send request to peer c5a3e756029444b2bf66ee0044f5bc35 for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.112.255:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 130 times. W0109 07:20:59.994902 1578 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer c5a3e756029444b2bf66ee0044f5bc35 (aserbin-kudu-jepsen-ts-1.gce.cloudera.com:7050): Couldn't send request to peer c5a3e756029444b2bf66ee0044f5bc35 for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.112.255:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 131 times. W0109 07:21:00.043141 1578 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer c5a3e756029444b2bf66ee0044f5bc35 (aserbin-kudu-jepsen-ts-1.gce.cloudera.com:7050): Couldn't send request to peer c5a3e756029444b2bf66ee0044f5bc35 for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.112.255:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 132 times. W0109 07:21:00.074347 1578 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer c5a3e756029444b2bf66ee0044f5bc35 (aserbin-kudu-jepsen-ts-1.gce.cloudera.com:7050): Couldn't send request to peer c5a3e756029444b2bf66ee0044f5bc35 for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.112.255:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 133 times. W0109 07:21:00.118479 1578 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer c5a3e756029444b2bf66ee0044f5bc35 (aserbin-kudu-jepsen-ts-1.gce.cloudera.com:7050): Couldn't send request to peer c5a3e756029444b2bf66ee0044f5bc35 for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.112.255:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 134 times. W0109 07:21:00.159780 1578 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer c5a3e756029444b2bf66ee0044f5bc35 (aserbin-kudu-jepsen-ts-1.gce.cloudera.com:7050): Couldn't send request to peer c5a3e756029444b2bf66ee0044f5bc35 for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.112.255:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 135 times. W0109 07:21:00.185807 1578 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer c5a3e756029444b2bf66ee0044f5bc35 (aserbin-kudu-jepsen-ts-1.gce.cloudera.com:7050): Couldn't send request to peer c5a3e756029444b2bf66ee0044f5bc35 for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.112.255:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 136 times. W0109 07:21:00.213044 1578 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer c5a3e756029444b2bf66ee0044f5bc35 (aserbin-kudu-jepsen-ts-1.gce.cloudera.com:7050): Couldn't send request to peer c5a3e756029444b2bf66ee0044f5bc35 for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.112.255:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 137 times. W0109 07:21:00.246248 1578 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer c5a3e756029444b2bf66ee0044f5bc35 (aserbin-kudu-jepsen-ts-1.gce.cloudera.com:7050): Couldn't send request to peer c5a3e756029444b2bf66ee0044f5bc35 for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.112.255:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 138 times. W0109 07:21:00.295408 1578 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer c5a3e756029444b2bf66ee0044f5bc35 (aserbin-kudu-jepsen-ts-1.gce.cloudera.com:7050): Couldn't send request to peer c5a3e756029444b2bf66ee0044f5bc35 for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.112.255:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 139 times. W0109 07:21:00.343642 1578 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer c5a3e756029444b2bf66ee0044f5bc35 (aserbin-kudu-jepsen-ts-1.gce.cloudera.com:7050): Couldn't send request to peer c5a3e756029444b2bf66ee0044f5bc35 for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.112.255:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 140 times. W0109 07:21:00.350565 1581 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer 7897578c54854ba0a5af32eb2aa04704 (aserbin-kudu-jepsen-ts-3.gce.cloudera.com:7050): Couldn't send request to peer 7897578c54854ba0a5af32eb2aa04704 for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.114.37:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 1 times. W0109 07:21:00.375794 1578 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer c5a3e756029444b2bf66ee0044f5bc35 (aserbin-kudu-jepsen-ts-1.gce.cloudera.com:7050): Couldn't send request to peer c5a3e756029444b2bf66ee0044f5bc35 for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.112.255:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 141 times. W0109 07:21:00.400689 1581 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer 7897578c54854ba0a5af32eb2aa04704 (aserbin-kudu-jepsen-ts-3.gce.cloudera.com:7050): Couldn't send request to peer 7897578c54854ba0a5af32eb2aa04704 for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.114.37:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 2 times. W0109 07:21:00.403164 1578 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer c5a3e756029444b2bf66ee0044f5bc35 (aserbin-kudu-jepsen-ts-1.gce.cloudera.com:7050): Couldn't send request to peer c5a3e756029444b2bf66ee0044f5bc35 for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.112.255:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 142 times. W0109 07:21:00.445468 1578 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer c5a3e756029444b2bf66ee0044f5bc35 (aserbin-kudu-jepsen-ts-1.gce.cloudera.com:7050): Couldn't send request to peer c5a3e756029444b2bf66ee0044f5bc35 for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.112.255:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 143 times. W0109 07:21:00.450798 1581 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer 7897578c54854ba0a5af32eb2aa04704 (aserbin-kudu-jepsen-ts-3.gce.cloudera.com:7050): Couldn't send request to peer 7897578c54854ba0a5af32eb2aa04704 for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.114.37:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 3 times. W0109 07:21:00.476936 1581 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer 7897578c54854ba0a5af32eb2aa04704 (aserbin-kudu-jepsen-ts-3.gce.cloudera.com:7050): Couldn't send request to peer 7897578c54854ba0a5af32eb2aa04704 for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.114.37:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 4 times. W0109 07:21:00.478569 1578 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer c5a3e756029444b2bf66ee0044f5bc35 (aserbin-kudu-jepsen-ts-1.gce.cloudera.com:7050): Couldn't send request to peer c5a3e756029444b2bf66ee0044f5bc35 for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.112.255:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 144 times. W0109 07:21:00.510767 1578 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer c5a3e756029444b2bf66ee0044f5bc35 (aserbin-kudu-jepsen-ts-1.gce.cloudera.com:7050): Couldn't send request to peer c5a3e756029444b2bf66ee0044f5bc35 for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.112.255:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 145 times. W0109 07:21:00.512989 1581 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer 7897578c54854ba0a5af32eb2aa04704 (aserbin-kudu-jepsen-ts-3.gce.cloudera.com:7050): Couldn't send request to peer 7897578c54854ba0a5af32eb2aa04704 for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.114.37:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 5 times. W0109 07:21:00.532812 1581 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer 591b1a1d383d4b659d7549653f4f136e (aserbin-kudu-jepsen-ts-2.gce.cloudera.com:7050): Couldn't send request to peer 591b1a1d383d4b659d7549653f4f136e for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.113.2:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 1 times. W0109 07:21:00.545994 1578 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer c5a3e756029444b2bf66ee0044f5bc35 (aserbin-kudu-jepsen-ts-1.gce.cloudera.com:7050): Couldn't send request to peer c5a3e756029444b2bf66ee0044f5bc35 for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.112.255:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 146 times. W0109 07:21:00.560209 1581 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer 7897578c54854ba0a5af32eb2aa04704 (aserbin-kudu-jepsen-ts-3.gce.cloudera.com:7050): Couldn't send request to peer 7897578c54854ba0a5af32eb2aa04704 for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.114.37:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 6 times. W0109 07:21:00.561822 1581 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer 591b1a1d383d4b659d7549653f4f136e (aserbin-kudu-jepsen-ts-2.gce.cloudera.com:7050): Couldn't send request to peer 591b1a1d383d4b659d7549653f4f136e for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.113.2:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 2 times. W0109 07:21:00.586300 1578 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer c5a3e756029444b2bf66ee0044f5bc35 (aserbin-kudu-jepsen-ts-1.gce.cloudera.com:7050): Couldn't send request to peer c5a3e756029444b2bf66ee0044f5bc35 for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.112.255:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 147 times. W0109 07:21:00.590338 1581 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer 7897578c54854ba0a5af32eb2aa04704 (aserbin-kudu-jepsen-ts-3.gce.cloudera.com:7050): Couldn't send request to peer 7897578c54854ba0a5af32eb2aa04704 for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.114.37:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 7 times. W0109 07:21:00.593968 1581 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer 591b1a1d383d4b659d7549653f4f136e (aserbin-kudu-jepsen-ts-2.gce.cloudera.com:7050): Couldn't send request to peer 591b1a1d383d4b659d7549653f4f136e for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.113.2:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 3 times. W0109 07:21:00.615459 1581 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer 7897578c54854ba0a5af32eb2aa04704 (aserbin-kudu-jepsen-ts-3.gce.cloudera.com:7050): Couldn't send request to peer 7897578c54854ba0a5af32eb2aa04704 for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.114.37:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 8 times. W0109 07:21:00.628147 1581 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer 591b1a1d383d4b659d7549653f4f136e (aserbin-kudu-jepsen-ts-2.gce.cloudera.com:7050): Couldn't send request to peer 591b1a1d383d4b659d7549653f4f136e for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.113.2:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 4 times. W0109 07:21:00.630539 1578 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer c5a3e756029444b2bf66ee0044f5bc35 (aserbin-kudu-jepsen-ts-1.gce.cloudera.com:7050): Couldn't send request to peer c5a3e756029444b2bf66ee0044f5bc35 for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.112.255:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 148 times. W0109 07:21:00.640651 1581 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer 7897578c54854ba0a5af32eb2aa04704 (aserbin-kudu-jepsen-ts-3.gce.cloudera.com:7050): Couldn't send request to peer 7897578c54854ba0a5af32eb2aa04704 for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.114.37:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 9 times. W0109 07:21:00.664379 1581 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer 591b1a1d383d4b659d7549653f4f136e (aserbin-kudu-jepsen-ts-2.gce.cloudera.com:7050): Couldn't send request to peer 591b1a1d383d4b659d7549653f4f136e for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.113.2:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 5 times. W0109 07:21:00.670771 1581 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer 7897578c54854ba0a5af32eb2aa04704 (aserbin-kudu-jepsen-ts-3.gce.cloudera.com:7050): Couldn't send request to peer 7897578c54854ba0a5af32eb2aa04704 for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.114.37:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 10 times. W0109 07:21:00.677834 1578 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer c5a3e756029444b2bf66ee0044f5bc35 (aserbin-kudu-jepsen-ts-1.gce.cloudera.com:7050): Couldn't send request to peer c5a3e756029444b2bf66ee0044f5bc35 for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.112.255:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 149 times. W0109 07:21:00.707125 1581 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer 7897578c54854ba0a5af32eb2aa04704 (aserbin-kudu-jepsen-ts-3.gce.cloudera.com:7050): Couldn't send request to peer 7897578c54854ba0a5af32eb2aa04704 for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.114.37:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 11 times. W0109 07:21:00.713474 1581 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer 591b1a1d383d4b659d7549653f4f136e (aserbin-kudu-jepsen-ts-2.gce.cloudera.com:7050): Couldn't send request to peer 591b1a1d383d4b659d7549653f4f136e for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.113.2:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 6 times. W0109 07:21:00.714093 1578 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer c5a3e756029444b2bf66ee0044f5bc35 (aserbin-kudu-jepsen-ts-1.gce.cloudera.com:7050): Couldn't send request to peer c5a3e756029444b2bf66ee0044f5bc35 for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.112.255:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 150 times. I0109 07:21:06.642036 1743 tablet_server_main.cc:56] Tablet server non-default flags: --raft_heartbeat_interval_ms=50 --fs_data_dirs=/var/lib/kudu/tserver --fs_wal_dir=/var/lib/kudu/tserver --tserver_master_addrs=aserbin-kudu-jepsen-master.gce.cloudera.com:7051 --heap_profile_path=/tmp/kudu-tserver.1743 --unlock_experimental_flags=true --unlock_unsafe_flags=true --flagfile=/etc/kudu/conf/tserver.gflagfile --log_dir=/var/log/kudu --logtostderr=true Tablet server version: kudu 1.3.0-SNAPSHOT revision 3a23b058af0f4de04f8827255de405fff7bf9ee1 build type RELEASE built by jenkins at 09 Jan 2017 06:07:18 PST on aserbin-kudu-jepsen-control-node-debian8.gce.cloudera.com build id 2017-01-09_06-07-06 I0109 07:21:06.642206 1743 mem_tracker.cc:140] MemTracker: hard memory limit is 0.464404 GB I0109 07:21:06.642218 1743 mem_tracker.cc:142] MemTracker: soft memory limit is 0.278643 GB I0109 07:21:06.643432 1743 tablet_server_main.cc:63] Initializing tablet server... I0109 07:21:06.694387 1743 hybrid_clock.cc:177] HybridClock initialized. Resolution in nanos?: 1 Wait times tolerance adjustment: 1.0005 Current error: 394013 I0109 07:21:06.695171 1743 env_posix.cc:1286] Not raising process file limit of 65536; it is already as high as it can go I0109 07:21:06.695206 1743 file_cache.cc:401] Constructed file cache lbm with capacity 32768 I0109 07:21:06.696270 1743 fs_manager.cc:251] Opened local filesystem: /var/lib/kudu/tserver uuid: "710c236995db46699eb9055c19ac506e" format_stamp: "Formatted at 2017-01-09 15:20:10 on aserbin-kudu-jepsen-ts-4" I0109 07:21:06.699396 1743 ts_tablet_manager.cc:701] Loading metadata for tablet 9e99b710634a44f4a948132a5a180600 I0109 07:21:06.699666 1743 ts_tablet_manager.cc:863] Registered tablet 9e99b710634a44f4a948132a5a180600 I0109 07:21:06.699851 1757 ts_tablet_manager.cc:725] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e: Bootstrapping tablet I0109 07:21:06.700098 1757 tablet_bootstrap.cc:381] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e: Bootstrap starting. I0109 07:21:06.700415 1757 tablet_bootstrap.cc:540] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e: Time spent opening tablet: real 0.000s user 0.000s sys 0.000s I0109 07:21:06.700518 1757 tablet_bootstrap.cc:596] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e: Will attempt to recover log segment /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600/wal-000000001 to /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery/wal-000000001 I0109 07:21:06.700557 1757 tablet_bootstrap.cc:604] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e: Moving log directory /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600 to recovery directory /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery in preparation for log replay I0109 07:21:06.700765 1757 log_util.cc:315] Log segment /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery/wal-000000001 has no footer. This segment was likely being written when the server previously shut down. I0109 07:21:06.700778 1757 log_reader.cc:160] Log segment /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery/wal-000000001 was likely left in-progress after a previous crash. Will try to rebuild footer by scanning data. I0109 07:21:06.702214 1743 tablet_server_main.cc:66] Starting tablet server... I0109 07:21:06.704774 1757 log_util.cc:569] Scanning /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery/wal-000000001 for valid entry headers following offset 36828... I0109 07:21:06.775909 1757 log_util.cc:606] Found no log entry headers I0109 07:21:06.775986 1757 log_util.cc:218] Ignoring log segment corruption in /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery/wal-000000001 because there are no log entries following the corrupted one. The server probably crashed in the middle of writing an entry to the write-ahead log or downloaded an active log via tablet copy. Error detail: Corruption: CRC mismatch in log entry header: Log file corruption detected. Failed trying to read batch #0 at offset 36816 for log segment /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery/wal-000000001: Prior entries: [off=36385 COMMIT (5.159)] [off=36583 REPLICATE (5.160)] [off=36618 COMMIT (5.160)] [off=36816 REPLICATE (5.161)] I0109 07:21:06.776053 1757 log_util.cc:368] Successfully rebuilt footer for segment: /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery/wal-000000001 (valid entries through byte offset 36816) I0109 07:21:06.776124 1757 tablet.cc:983] T 9e99b710634a44f4a948132a5a180600 Rewinding schema during bootstrap to Schema [ 0:key[string NOT NULL], 1:value[int32 NOT NULL] ] I0109 07:21:06.776366 1757 log.cc:351] Log is configured to *not* fsync() on all Append() calls I0109 07:21:06.782049 1757 tablet_bootstrap.cc:810] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e: Overwriting operations starting at: 2.50 up to: 2.50 with operation: 4.50 I0109 07:21:06.787969 1757 tablet_bootstrap.cc:381] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e: Bootstrap replayed 1/1 log segments. Stats: ops{read=162 overwritten=1 applied=160 ignored=0} inserts{seen=157 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates I0109 07:21:06.788154 1757 tablet_bootstrap.cc:1017] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e: ReplayState: Previous OpId: 5.161, Committed OpId: 5.160, Pending Replicates: 1, Pending Commits: 0 I0109 07:21:06.788166 1757 tablet_bootstrap.cc:1017] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e: Dumping REPLICATES: I0109 07:21:06.788172 1757 tablet_bootstrap.cc:1017] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e: type: REPLICATE replicate { id { term: 5 index: 161 } timestamp: 6078362667140030464 op_type: WRITE_OP write_request { tablet_id: "9e99b710634a44f4a948132a5a180600" schema { columns { name: "key" type: STRING is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "value" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } row_operations { rows: "\005\003\000\000\000\000\000\000\000\000\001\000\000\000\000\000\000\000\007\000\000\000" indirect_data: "x" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 6078362665418371072 } request_id { client_id: "a89de875ce2044048caa38631f674301" seq_no: 32 first_incomplete_seq_no: 32 attempt_no: 1 } } I0109 07:21:06.788555 1757 tablet_bootstrap.cc:633] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e: Preparing to delete log recovery files and directory /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery I0109 07:21:06.788568 1757 tablet_bootstrap.cc:636] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e: Renaming log recovery dir from /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery to /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery-1483975266788567 I0109 07:21:06.788596 1757 tablet_bootstrap.cc:646] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e: Deleting all files from renamed log recovery directory /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery-1483975266788567 I0109 07:21:06.788738 1757 tablet_bootstrap.cc:649] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e: Completed deletion of old log recovery files and directory /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery-1483975266788567 I0109 07:21:06.788750 1757 tablet_bootstrap.cc:381] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e: Bootstrap complete. I0109 07:21:06.793248 1757 ts_tablet_manager.cc:730] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e: Time spent bootstrapping tablet: real 0.093s user 0.016s sys 0.036s I0109 07:21:06.793740 1757 raft_consensus.cc:287] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [term 5 FOLLOWER]: Replica starting. Triggering 1 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "710c236995db46699eb9055c19ac506e" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-4.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "1678276a36f849c28459de6b6788965a" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-5.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "591b1a1d383d4b659d7549653f4f136e" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-2.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "c5a3e756029444b2bf66ee0044f5bc35" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-1.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "7897578c54854ba0a5af32eb2aa04704" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-3.gce.cloudera.com" port: 7050 } } I0109 07:21:06.793980 1757 raft_consensus.cc:547] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [term 5 FOLLOWER]: Becoming Follower/Learner. State: Replica: 710c236995db46699eb9055c19ac506e, State: 1, Role: FOLLOWER I0109 07:21:06.794000 1757 consensus_queue.cc:175] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 160, Last appended: 5.161, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER I0109 07:21:06.794244 1757 ts_tablet_manager.cc:757] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s I0109 07:21:06.801407 1743 rpc_server.cc:164] RPC server started. Bound to: 0.0.0.0:7050 I0109 07:21:06.801483 1743 webserver.cc:126] Starting webserver on 0.0.0.0:8050 I0109 07:21:06.801492 1743 webserver.cc:137] Document root disabled I0109 07:21:06.801643 1743 webserver.cc:221] Webserver started. Bound to: http://0.0.0.0:8050/ I0109 07:21:06.802016 1743 tablet_server_main.cc:69] Tablet server successfully started. I0109 07:21:06.854665 1874 heartbeater.cc:292] Connected to a master server at aserbin-kudu-jepsen-master.gce.cloudera.com:7051 I0109 07:21:06.854717 1874 heartbeater.cc:359] Registering TS with master... I0109 07:21:06.854780 1874 heartbeater.cc:373] Master aserbin-kudu-jepsen-master.gce.cloudera.com:7051 requested a full tablet report, sending... I0109 07:21:06.977129 1798 raft_consensus.cc:410] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0109 07:21:06.977188 1798 raft_consensus.cc:2158] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [term 5 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 0.188s I0109 07:21:06.977258 1798 raft_consensus.cc:434] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "710c236995db46699eb9055c19ac506e" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-4.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "1678276a36f849c28459de6b6788965a" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-5.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "591b1a1d383d4b659d7549653f4f136e" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-2.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "c5a3e756029444b2bf66ee0044f5bc35" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-1.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "7897578c54854ba0a5af32eb2aa04704" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-3.gce.cloudera.com" port: 7050 } } I0109 07:21:07.173569 1798 leader_election.cc:215] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [CANDIDATE]: Term 6 pre-election: Requesting pre-vote from peer 1678276a36f849c28459de6b6788965a I0109 07:21:07.173831 1831 raft_consensus.cc:1691] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7897578c54854ba0a5af32eb2aa04704 for term 6 because replica has last-logged OpId of term: 5 index: 161, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 158. W0109 07:21:07.173969 1831 rpcz_store.cc:234] Call kudu.consensus.ConsensusService.RequestConsensusVote from 172.31.114.37:47218 (request call id 1) took 167ms (client timeout 156). W0109 07:21:07.174017 1831 rpcz_store.cc:238] Trace: 0109 07:21:07.006397 (+ 0us) service_pool.cc:143] Inserting onto call queue 0109 07:21:07.006409 (+ 12us) service_pool.cc:202] Handling call 0109 07:21:07.173946 (+167537us) spinlock_profiling.cc:255] Waited 167 ms on lock 0x33a91b0. stack: 0000000001a20b21 00000000009d8b77 00000000007e48af 0000000000a8971f 0000000000a8a342 0000000001a2dc19 00007f53424960a3 00007f5340d2687c ffffffffffffffff 0109 07:21:07.173960 (+ 14us) inbound_call.cc:130] Queueing success response Metrics: {"spinlock_wait_cycles":418376576} I0109 07:21:07.174091 1798 leader_election.cc:215] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [CANDIDATE]: Term 6 pre-election: Requesting pre-vote from peer 591b1a1d383d4b659d7549653f4f136e I0109 07:21:07.174206 1798 leader_election.cc:215] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [CANDIDATE]: Term 6 pre-election: Requesting pre-vote from peer c5a3e756029444b2bf66ee0044f5bc35 I0109 07:21:07.174283 1798 leader_election.cc:215] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [CANDIDATE]: Term 6 pre-election: Requesting pre-vote from peer 7897578c54854ba0a5af32eb2aa04704 W0109 07:21:07.175112 1755 leader_election.cc:271] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 591b1a1d383d4b659d7549653f4f136e: Network error: Client connection negotiation failed: client connection to 172.31.113.2:7050: connect: Connection refused (error 111) W0109 07:21:07.175161 1753 leader_election.cc:271] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 1678276a36f849c28459de6b6788965a: Network error: Client connection negotiation failed: client connection to 172.31.114.39:7050: connect: Connection refused (error 111) I0109 07:21:07.175997 1755 leader_election.cc:347] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [CANDIDATE]: Term 6 pre-election: Vote granted by peer 7897578c54854ba0a5af32eb2aa04704 I0109 07:21:07.176822 1752 leader_election.cc:347] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [CANDIDATE]: Term 6 pre-election: Vote granted by peer c5a3e756029444b2bf66ee0044f5bc35 I0109 07:21:07.176839 1752 leader_election.cc:242] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. I0109 07:21:07.176995 1880 raft_consensus.cc:2158] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [term 5 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 0.161s I0109 07:21:07.177033 1880 raft_consensus.cc:1968] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [term 5 FOLLOWER]: Leader pre-election won for term 6 I0109 07:21:07.177042 1880 raft_consensus.cc:410] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0109 07:21:07.177052 1880 raft_consensus.cc:2158] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [term 5 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 0.181s I0109 07:21:07.177057 1880 raft_consensus.cc:2202] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [term 5 FOLLOWER]: Advancing to term 6 I0109 07:21:07.177397 1880 raft_consensus.cc:434] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "710c236995db46699eb9055c19ac506e" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-4.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "1678276a36f849c28459de6b6788965a" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-5.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "591b1a1d383d4b659d7549653f4f136e" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-2.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "c5a3e756029444b2bf66ee0044f5bc35" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-1.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "7897578c54854ba0a5af32eb2aa04704" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-3.gce.cloudera.com" port: 7050 } } I0109 07:21:07.373360 1880 leader_election.cc:215] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [CANDIDATE]: Term 6 election: Requesting vote from peer 1678276a36f849c28459de6b6788965a I0109 07:21:07.373666 1880 leader_election.cc:215] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [CANDIDATE]: Term 6 election: Requesting vote from peer 591b1a1d383d4b659d7549653f4f136e I0109 07:21:07.373916 1880 leader_election.cc:215] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [CANDIDATE]: Term 6 election: Requesting vote from peer c5a3e756029444b2bf66ee0044f5bc35 W0109 07:21:07.373975 1755 leader_election.cc:271] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer 591b1a1d383d4b659d7549653f4f136e: Network error: Client connection negotiation failed: client connection to 172.31.113.2:7050: connect: Connection refused (error 111) I0109 07:21:07.374058 1880 leader_election.cc:215] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [CANDIDATE]: Term 6 election: Requesting vote from peer 7897578c54854ba0a5af32eb2aa04704 I0109 07:21:07.375074 1752 leader_election.cc:347] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [CANDIDATE]: Term 6 election: Vote granted by peer c5a3e756029444b2bf66ee0044f5bc35 I0109 07:21:07.375103 1755 leader_election.cc:347] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [CANDIDATE]: Term 6 election: Vote granted by peer 7897578c54854ba0a5af32eb2aa04704 I0109 07:21:07.375111 1755 leader_election.cc:242] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. I0109 07:21:07.375154 1880 raft_consensus.cc:2158] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [term 6 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 0.208s I0109 07:21:07.375167 1880 raft_consensus.cc:1968] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [term 6 FOLLOWER]: Leader election won for term 6 I0109 07:21:07.375196 1880 raft_consensus.cc:517] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [term 6 LEADER]: Becoming Leader. State: Replica: 710c236995db46699eb9055c19ac506e, State: 1, Role: LEADER I0109 07:21:07.375293 1880 consensus_queue.cc:157] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 160, Committed index: 160, Last appended: 5.161, Current term: 6, Majority size: 3, State: 1, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "710c236995db46699eb9055c19ac506e" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-4.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "1678276a36f849c28459de6b6788965a" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-5.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "591b1a1d383d4b659d7549653f4f136e" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-2.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "c5a3e756029444b2bf66ee0044f5bc35" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-1.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "7897578c54854ba0a5af32eb2aa04704" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-3.gce.cloudera.com" port: 7050 } } I0109 07:21:07.375632 1753 leader_election.cc:347] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [CANDIDATE]: Term 6 election: Vote granted by peer 1678276a36f849c28459de6b6788965a I0109 07:21:07.572204 1874 heartbeater.cc:365] Master aserbin-kudu-jepsen-master.gce.cloudera.com:7051 was elected leader, sending a full tablet report... I0109 07:21:07.572998 1880 consensus_queue.cc:693] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [LEADER]: Connected to new peer: Peer: 1678276a36f849c28459de6b6788965a, Is new: false, Last received: 5.161, Next index: 162, Last known committed idx: 160, Last exchange result: ERROR, Needs tablet copy: false I0109 07:21:07.573467 1880 consensus_queue.cc:693] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [LEADER]: Connected to new peer: Peer: c5a3e756029444b2bf66ee0044f5bc35, Is new: false, Last received: 5.129, Next index: 130, Last known committed idx: 129, Last exchange result: ERROR, Needs tablet copy: false I0109 07:21:07.575063 1880 consensus_queue.cc:693] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [LEADER]: Connected to new peer: Peer: 591b1a1d383d4b659d7549653f4f136e, Is new: false, Last received: 5.160, Next index: 161, Last known committed idx: 160, Last exchange result: ERROR, Needs tablet copy: false I0109 07:21:07.576283 1880 consensus_queue.cc:693] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [LEADER]: Connected to new peer: Peer: 7897578c54854ba0a5af32eb2aa04704, Is new: false, Last received: 5.158, Next index: 159, Last known committed idx: 158, Last exchange result: ERROR, Needs tablet copy: false W0109 07:21:13.577644 1755 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer 7897578c54854ba0a5af32eb2aa04704 (aserbin-kudu-jepsen-ts-3.gce.cloudera.com:7050): Couldn't send request to peer 7897578c54854ba0a5af32eb2aa04704 for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.114.37:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 1 times. W0109 07:21:13.617537 1755 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer 7897578c54854ba0a5af32eb2aa04704 (aserbin-kudu-jepsen-ts-3.gce.cloudera.com:7050): Couldn't send request to peer 7897578c54854ba0a5af32eb2aa04704 for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.114.37:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 2 times. W0109 07:21:13.655702 1755 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer 7897578c54854ba0a5af32eb2aa04704 (aserbin-kudu-jepsen-ts-3.gce.cloudera.com:7050): Couldn't send request to peer 7897578c54854ba0a5af32eb2aa04704 for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.114.37:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 3 times. W0109 07:21:13.683019 1755 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer 7897578c54854ba0a5af32eb2aa04704 (aserbin-kudu-jepsen-ts-3.gce.cloudera.com:7050): Couldn't send request to peer 7897578c54854ba0a5af32eb2aa04704 for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.114.37:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 4 times. W0109 07:21:13.728096 1755 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer 7897578c54854ba0a5af32eb2aa04704 (aserbin-kudu-jepsen-ts-3.gce.cloudera.com:7050): Couldn't send request to peer 7897578c54854ba0a5af32eb2aa04704 for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.114.37:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 5 times. W0109 07:21:13.763247 1755 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer 7897578c54854ba0a5af32eb2aa04704 (aserbin-kudu-jepsen-ts-3.gce.cloudera.com:7050): Couldn't send request to peer 7897578c54854ba0a5af32eb2aa04704 for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.114.37:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 6 times. W0109 07:21:13.790426 1755 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer 7897578c54854ba0a5af32eb2aa04704 (aserbin-kudu-jepsen-ts-3.gce.cloudera.com:7050): Couldn't send request to peer 7897578c54854ba0a5af32eb2aa04704 for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.114.37:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 7 times. W0109 07:21:13.817827 1753 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer 1678276a36f849c28459de6b6788965a (aserbin-kudu-jepsen-ts-5.gce.cloudera.com:7050): Couldn't send request to peer 1678276a36f849c28459de6b6788965a for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.114.39:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 1 times. W0109 07:21:13.835546 1755 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer 7897578c54854ba0a5af32eb2aa04704 (aserbin-kudu-jepsen-ts-3.gce.cloudera.com:7050): Couldn't send request to peer 7897578c54854ba0a5af32eb2aa04704 for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.114.37:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 8 times. W0109 07:21:13.867952 1753 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer 1678276a36f849c28459de6b6788965a (aserbin-kudu-jepsen-ts-5.gce.cloudera.com:7050): Couldn't send request to peer 1678276a36f849c28459de6b6788965a for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.114.39:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 2 times. W0109 07:21:13.869593 1755 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer 7897578c54854ba0a5af32eb2aa04704 (aserbin-kudu-jepsen-ts-3.gce.cloudera.com:7050): Couldn't send request to peer 7897578c54854ba0a5af32eb2aa04704 for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.114.37:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 9 times. W0109 07:21:13.911998 1755 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer 7897578c54854ba0a5af32eb2aa04704 (aserbin-kudu-jepsen-ts-3.gce.cloudera.com:7050): Couldn't send request to peer 7897578c54854ba0a5af32eb2aa04704 for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.114.37:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 10 times. W0109 07:21:13.917953 1753 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer 1678276a36f849c28459de6b6788965a (aserbin-kudu-jepsen-ts-5.gce.cloudera.com:7050): Couldn't send request to peer 1678276a36f849c28459de6b6788965a for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.114.39:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 3 times. W0109 07:21:13.959056 1755 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer 7897578c54854ba0a5af32eb2aa04704 (aserbin-kudu-jepsen-ts-3.gce.cloudera.com:7050): Couldn't send request to peer 7897578c54854ba0a5af32eb2aa04704 for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.114.37:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 11 times. W0109 07:21:13.968082 1753 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer 1678276a36f849c28459de6b6788965a (aserbin-kudu-jepsen-ts-5.gce.cloudera.com:7050): Couldn't send request to peer 1678276a36f849c28459de6b6788965a for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.114.39:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 4 times. W0109 07:21:13.989245 1755 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer 7897578c54854ba0a5af32eb2aa04704 (aserbin-kudu-jepsen-ts-3.gce.cloudera.com:7050): Couldn't send request to peer 7897578c54854ba0a5af32eb2aa04704 for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.114.37:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 12 times. W0109 07:21:14.018244 1753 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer 1678276a36f849c28459de6b6788965a (aserbin-kudu-jepsen-ts-5.gce.cloudera.com:7050): Couldn't send request to peer 1678276a36f849c28459de6b6788965a for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.114.39:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 5 times. W0109 07:21:14.032481 1755 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer 7897578c54854ba0a5af32eb2aa04704 (aserbin-kudu-jepsen-ts-3.gce.cloudera.com:7050): Couldn't send request to peer 7897578c54854ba0a5af32eb2aa04704 for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.114.37:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 13 times. W0109 07:21:14.068284 1753 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer 1678276a36f849c28459de6b6788965a (aserbin-kudu-jepsen-ts-5.gce.cloudera.com:7050): Couldn't send request to peer 1678276a36f849c28459de6b6788965a for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.114.39:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 6 times. W0109 07:21:14.069335 1752 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer c5a3e756029444b2bf66ee0044f5bc35 (aserbin-kudu-jepsen-ts-1.gce.cloudera.com:7050): Couldn't send request to peer c5a3e756029444b2bf66ee0044f5bc35 for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.112.255:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 1 times. W0109 07:21:14.069547 1755 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer 7897578c54854ba0a5af32eb2aa04704 (aserbin-kudu-jepsen-ts-3.gce.cloudera.com:7050): Couldn't send request to peer 7897578c54854ba0a5af32eb2aa04704 for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.114.37:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 14 times. W0109 07:21:14.118567 1753 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer 1678276a36f849c28459de6b6788965a (aserbin-kudu-jepsen-ts-5.gce.cloudera.com:7050): Couldn't send request to peer 1678276a36f849c28459de6b6788965a for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.114.39:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 7 times. W0109 07:21:14.118732 1755 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer 7897578c54854ba0a5af32eb2aa04704 (aserbin-kudu-jepsen-ts-3.gce.cloudera.com:7050): Couldn't send request to peer 7897578c54854ba0a5af32eb2aa04704 for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.114.37:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 15 times. W0109 07:21:14.119307 1752 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer c5a3e756029444b2bf66ee0044f5bc35 (aserbin-kudu-jepsen-ts-1.gce.cloudera.com:7050): Couldn't send request to peer c5a3e756029444b2bf66ee0044f5bc35 for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.112.255:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 2 times. W0109 07:21:14.156896 1753 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer 1678276a36f849c28459de6b6788965a (aserbin-kudu-jepsen-ts-5.gce.cloudera.com:7050): Couldn't send request to peer 1678276a36f849c28459de6b6788965a for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.114.39:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 8 times. W0109 07:21:14.159842 1755 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer 7897578c54854ba0a5af32eb2aa04704 (aserbin-kudu-jepsen-ts-3.gce.cloudera.com:7050): Couldn't send request to peer 7897578c54854ba0a5af32eb2aa04704 for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.114.37:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 16 times. W0109 07:21:14.162418 1752 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer c5a3e756029444b2bf66ee0044f5bc35 (aserbin-kudu-jepsen-ts-1.gce.cloudera.com:7050): Couldn't send request to peer c5a3e756029444b2bf66ee0044f5bc35 for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.112.255:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 3 times. W0109 07:21:14.185968 1755 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer 7897578c54854ba0a5af32eb2aa04704 (aserbin-kudu-jepsen-ts-3.gce.cloudera.com:7050): Couldn't send request to peer 7897578c54854ba0a5af32eb2aa04704 for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.114.37:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 17 times. W0109 07:21:14.206373 1753 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer 1678276a36f849c28459de6b6788965a (aserbin-kudu-jepsen-ts-5.gce.cloudera.com:7050): Couldn't send request to peer 1678276a36f849c28459de6b6788965a for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.114.39:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 9 times. W0109 07:21:14.209570 1752 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer c5a3e756029444b2bf66ee0044f5bc35 (aserbin-kudu-jepsen-ts-1.gce.cloudera.com:7050): Couldn't send request to peer c5a3e756029444b2bf66ee0044f5bc35 for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.112.255:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 4 times. W0109 07:21:14.227177 1755 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer 7897578c54854ba0a5af32eb2aa04704 (aserbin-kudu-jepsen-ts-3.gce.cloudera.com:7050): Couldn't send request to peer 7897578c54854ba0a5af32eb2aa04704 for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.114.37:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 18 times. W0109 07:21:14.237507 1753 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer 1678276a36f849c28459de6b6788965a (aserbin-kudu-jepsen-ts-5.gce.cloudera.com:7050): Couldn't send request to peer 1678276a36f849c28459de6b6788965a for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.114.39:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 10 times. W0109 07:21:14.237617 1752 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer c5a3e756029444b2bf66ee0044f5bc35 (aserbin-kudu-jepsen-ts-1.gce.cloudera.com:7050): Couldn't send request to peer c5a3e756029444b2bf66ee0044f5bc35 for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.112.255:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 5 times. W0109 07:21:14.265383 1755 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer 7897578c54854ba0a5af32eb2aa04704 (aserbin-kudu-jepsen-ts-3.gce.cloudera.com:7050): Couldn't send request to peer 7897578c54854ba0a5af32eb2aa04704 for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.114.37:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 19 times. W0109 07:21:14.273869 1752 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer c5a3e756029444b2bf66ee0044f5bc35 (aserbin-kudu-jepsen-ts-1.gce.cloudera.com:7050): Couldn't send request to peer c5a3e756029444b2bf66ee0044f5bc35 for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.112.255:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 6 times. W0109 07:21:14.283613 1753 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer 1678276a36f849c28459de6b6788965a (aserbin-kudu-jepsen-ts-5.gce.cloudera.com:7050): Couldn't send request to peer 1678276a36f849c28459de6b6788965a for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.114.39:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 11 times. W0109 07:21:14.285607 1755 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer 591b1a1d383d4b659d7549653f4f136e (aserbin-kudu-jepsen-ts-2.gce.cloudera.com:7050): Couldn't send request to peer 591b1a1d383d4b659d7549653f4f136e for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.113.2:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 1 times. W0109 07:21:14.305464 1755 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer 7897578c54854ba0a5af32eb2aa04704 (aserbin-kudu-jepsen-ts-3.gce.cloudera.com:7050): Couldn't send request to peer 7897578c54854ba0a5af32eb2aa04704 for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.114.37:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 20 times. W0109 07:21:14.311909 1753 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer 1678276a36f849c28459de6b6788965a (aserbin-kudu-jepsen-ts-5.gce.cloudera.com:7050): Couldn't send request to peer 1678276a36f849c28459de6b6788965a for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.114.39:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 12 times. W0109 07:21:14.315280 1752 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer c5a3e756029444b2bf66ee0044f5bc35 (aserbin-kudu-jepsen-ts-1.gce.cloudera.com:7050): Couldn't send request to peer c5a3e756029444b2bf66ee0044f5bc35 for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.112.255:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 7 times. W0109 07:21:14.335834 1755 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer 591b1a1d383d4b659d7549653f4f136e (aserbin-kudu-jepsen-ts-2.gce.cloudera.com:7050): Couldn't send request to peer 591b1a1d383d4b659d7549653f4f136e for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.113.2:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 2 times. W0109 07:21:14.342380 1753 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer 1678276a36f849c28459de6b6788965a (aserbin-kudu-jepsen-ts-5.gce.cloudera.com:7050): Couldn't send request to peer 1678276a36f849c28459de6b6788965a for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.114.39:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 13 times. W0109 07:21:14.344789 1755 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer 7897578c54854ba0a5af32eb2aa04704 (aserbin-kudu-jepsen-ts-3.gce.cloudera.com:7050): Couldn't send request to peer 7897578c54854ba0a5af32eb2aa04704 for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.114.37:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 21 times. W0109 07:21:14.356873 1752 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer c5a3e756029444b2bf66ee0044f5bc35 (aserbin-kudu-jepsen-ts-1.gce.cloudera.com:7050): Couldn't send request to peer c5a3e756029444b2bf66ee0044f5bc35 for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.112.255:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 8 times. W0109 07:21:14.383119 1755 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer 7897578c54854ba0a5af32eb2aa04704 (aserbin-kudu-jepsen-ts-3.gce.cloudera.com:7050): Couldn't send request to peer 7897578c54854ba0a5af32eb2aa04704 for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.114.37:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 22 times. W0109 07:21:14.384305 1753 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer 1678276a36f849c28459de6b6788965a (aserbin-kudu-jepsen-ts-5.gce.cloudera.com:7050): Couldn't send request to peer 1678276a36f849c28459de6b6788965a for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.114.39:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 14 times. W0109 07:21:14.385756 1755 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer 591b1a1d383d4b659d7549653f4f136e (aserbin-kudu-jepsen-ts-2.gce.cloudera.com:7050): Couldn't send request to peer 591b1a1d383d4b659d7549653f4f136e for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.113.2:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 3 times. W0109 07:21:14.388084 1752 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer c5a3e756029444b2bf66ee0044f5bc35 (aserbin-kudu-jepsen-ts-1.gce.cloudera.com:7050): Couldn't send request to peer c5a3e756029444b2bf66ee0044f5bc35 for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.112.255:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 9 times. W0109 07:21:14.419541 1755 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer 7897578c54854ba0a5af32eb2aa04704 (aserbin-kudu-jepsen-ts-3.gce.cloudera.com:7050): Couldn't send request to peer 7897578c54854ba0a5af32eb2aa04704 for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.114.37:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 23 times. W0109 07:21:14.429476 1752 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer c5a3e756029444b2bf66ee0044f5bc35 (aserbin-kudu-jepsen-ts-1.gce.cloudera.com:7050): Couldn't send request to peer c5a3e756029444b2bf66ee0044f5bc35 for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.112.255:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 10 times. W0109 07:21:14.433594 1753 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer 1678276a36f849c28459de6b6788965a (aserbin-kudu-jepsen-ts-5.gce.cloudera.com:7050): Couldn't send request to peer 1678276a36f849c28459de6b6788965a for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.114.39:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 15 times. W0109 07:21:14.435940 1755 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer 591b1a1d383d4b659d7549653f4f136e (aserbin-kudu-jepsen-ts-2.gce.cloudera.com:7050): Couldn't send request to peer 591b1a1d383d4b659d7549653f4f136e for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.113.2:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 4 times. W0109 07:21:14.445073 1755 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer 7897578c54854ba0a5af32eb2aa04704 (aserbin-kudu-jepsen-ts-3.gce.cloudera.com:7050): Couldn't send request to peer 7897578c54854ba0a5af32eb2aa04704 for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.114.37:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 24 times. W0109 07:21:14.462813 1752 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e -> Peer c5a3e756029444b2bf66ee0044f5bc35 (aserbin-kudu-jepsen-ts-1.gce.cloudera.com:7050): Couldn't send request to peer c5a3e756029444b2bf66ee0044f5bc35 for tablet 9e99b710634a44f4a948132a5a180600. Status: Network error: Client connection negotiation failed: client connection to 172.31.112.255:7050: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 11 times. I0109 07:21:25.654464 1916 tablet_server_main.cc:56] Tablet server non-default flags: --raft_heartbeat_interval_ms=50 --fs_data_dirs=/var/lib/kudu/tserver --fs_wal_dir=/var/lib/kudu/tserver --tserver_master_addrs=aserbin-kudu-jepsen-master.gce.cloudera.com:7051 --heap_profile_path=/tmp/kudu-tserver.1916 --unlock_experimental_flags=true --unlock_unsafe_flags=true --flagfile=/etc/kudu/conf/tserver.gflagfile --log_dir=/var/log/kudu --logtostderr=true Tablet server version: kudu 1.3.0-SNAPSHOT revision 3a23b058af0f4de04f8827255de405fff7bf9ee1 build type RELEASE built by jenkins at 09 Jan 2017 06:07:18 PST on aserbin-kudu-jepsen-control-node-debian8.gce.cloudera.com build id 2017-01-09_06-07-06 I0109 07:21:25.654610 1916 mem_tracker.cc:140] MemTracker: hard memory limit is 0.464404 GB I0109 07:21:25.654621 1916 mem_tracker.cc:142] MemTracker: soft memory limit is 0.278643 GB I0109 07:21:25.655719 1916 tablet_server_main.cc:63] Initializing tablet server... I0109 07:21:25.705202 1916 hybrid_clock.cc:177] HybridClock initialized. Resolution in nanos?: 1 Wait times tolerance adjustment: 1.0005 Current error: 403513 I0109 07:21:25.705971 1916 env_posix.cc:1286] Not raising process file limit of 65536; it is already as high as it can go I0109 07:21:25.706022 1916 file_cache.cc:401] Constructed file cache lbm with capacity 32768 I0109 07:21:25.707054 1916 fs_manager.cc:251] Opened local filesystem: /var/lib/kudu/tserver uuid: "710c236995db46699eb9055c19ac506e" format_stamp: "Formatted at 2017-01-09 15:20:10 on aserbin-kudu-jepsen-ts-4" I0109 07:21:25.710098 1916 ts_tablet_manager.cc:701] Loading metadata for tablet 9e99b710634a44f4a948132a5a180600 I0109 07:21:25.710304 1916 ts_tablet_manager.cc:863] Registered tablet 9e99b710634a44f4a948132a5a180600 I0109 07:21:25.710459 1930 ts_tablet_manager.cc:725] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e: Bootstrapping tablet I0109 07:21:25.710645 1930 tablet_bootstrap.cc:381] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e: Bootstrap starting. I0109 07:21:25.710898 1930 tablet_bootstrap.cc:540] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e: Time spent opening tablet: real 0.000s user 0.000s sys 0.000s I0109 07:21:25.710986 1930 tablet_bootstrap.cc:596] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e: Will attempt to recover log segment /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600/wal-000000001 to /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery/wal-000000001 I0109 07:21:25.711000 1930 tablet_bootstrap.cc:604] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e: Moving log directory /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600 to recovery directory /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery in preparation for log replay I0109 07:21:25.711182 1930 log_util.cc:315] Log segment /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery/wal-000000001 has no footer. This segment was likely being written when the server previously shut down. I0109 07:21:25.711194 1930 log_reader.cc:160] Log segment /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery/wal-000000001 was likely left in-progress after a previous crash. Will try to rebuild footer by scanning data. I0109 07:21:25.712716 1916 tablet_server_main.cc:66] Starting tablet server... I0109 07:21:25.714624 1930 log_util.cc:569] Scanning /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery/wal-000000001 for valid entry headers following offset 48078... I0109 07:21:25.784153 1930 log_util.cc:606] Found no log entry headers I0109 07:21:25.784214 1930 log_util.cc:218] Ignoring log segment corruption in /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery/wal-000000001 because there are no log entries following the corrupted one. The server probably crashed in the middle of writing an entry to the write-ahead log or downloaded an active log via tablet copy. Error detail: Corruption: CRC mismatch in log entry header: Log file corruption detected. Failed trying to read batch #0 at offset 48066 for log segment /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery/wal-000000001: Prior entries: [off=47635 COMMIT (6.208)] [off=47833 REPLICATE (6.209)] [off=47868 COMMIT (6.209)] [off=48066 REPLICATE (6.210)] I0109 07:21:25.784224 1930 log_util.cc:368] Successfully rebuilt footer for segment: /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery/wal-000000001 (valid entries through byte offset 48066) I0109 07:21:25.784284 1930 tablet.cc:983] T 9e99b710634a44f4a948132a5a180600 Rewinding schema during bootstrap to Schema [ 0:key[string NOT NULL], 1:value[int32 NOT NULL] ] I0109 07:21:25.784523 1930 log.cc:351] Log is configured to *not* fsync() on all Append() calls I0109 07:21:25.787786 1930 tablet_bootstrap.cc:810] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e: Overwriting operations starting at: 2.50 up to: 2.50 with operation: 4.50 I0109 07:21:25.799007 1930 tablet_bootstrap.cc:381] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e: Bootstrap replayed 1/1 log segments. Stats: ops{read=211 overwritten=1 applied=209 ignored=0} inserts{seen=205 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates I0109 07:21:25.799247 1930 tablet_bootstrap.cc:1017] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e: ReplayState: Previous OpId: 6.210, Committed OpId: 6.209, Pending Replicates: 1, Pending Commits: 0 I0109 07:21:25.799257 1930 tablet_bootstrap.cc:1017] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e: Dumping REPLICATES: I0109 07:21:25.799263 1930 tablet_bootstrap.cc:1017] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e: type: REPLICATE replicate { id { term: 6 index: 210 } timestamp: 6078362722740658176 op_type: WRITE_OP write_request { tablet_id: "9e99b710634a44f4a948132a5a180600" schema { columns { name: "key" type: STRING is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "value" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } row_operations { rows: "\005\003\000\000\000\000\000\000\000\000\001\000\000\000\000\000\000\000\004\000\000\000" indirect_data: "x" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 6078362720224337920 } request_id { client_id: "551b04fdc262441283af896460477398" seq_no: 41 first_incomplete_seq_no: 41 attempt_no: 1 } } I0109 07:21:25.799662 1930 tablet_bootstrap.cc:633] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e: Preparing to delete log recovery files and directory /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery I0109 07:21:25.799675 1930 tablet_bootstrap.cc:636] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e: Renaming log recovery dir from /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery to /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery-1483975285799674 I0109 07:21:25.799691 1930 tablet_bootstrap.cc:646] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e: Deleting all files from renamed log recovery directory /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery-1483975285799674 I0109 07:21:25.799809 1930 tablet_bootstrap.cc:649] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e: Completed deletion of old log recovery files and directory /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery-1483975285799674 I0109 07:21:25.799823 1930 tablet_bootstrap.cc:381] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e: Bootstrap complete. I0109 07:21:25.804420 1930 ts_tablet_manager.cc:730] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e: Time spent bootstrapping tablet: real 0.094s user 0.020s sys 0.032s I0109 07:21:25.804904 1930 raft_consensus.cc:287] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [term 6 FOLLOWER]: Replica starting. Triggering 1 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "710c236995db46699eb9055c19ac506e" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-4.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "1678276a36f849c28459de6b6788965a" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-5.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "591b1a1d383d4b659d7549653f4f136e" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-2.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "c5a3e756029444b2bf66ee0044f5bc35" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-1.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "7897578c54854ba0a5af32eb2aa04704" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-3.gce.cloudera.com" port: 7050 } } I0109 07:21:25.805155 1930 raft_consensus.cc:547] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [term 6 FOLLOWER]: Becoming Follower/Learner. State: Replica: 710c236995db46699eb9055c19ac506e, State: 1, Role: FOLLOWER I0109 07:21:25.805193 1930 consensus_queue.cc:175] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 209, Last appended: 6.210, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER I0109 07:21:25.805436 1930 ts_tablet_manager.cc:757] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s I0109 07:21:25.812764 1916 rpc_server.cc:164] RPC server started. Bound to: 0.0.0.0:7050 I0109 07:21:25.812866 1916 webserver.cc:126] Starting webserver on 0.0.0.0:8050 I0109 07:21:25.812875 1916 webserver.cc:137] Document root disabled I0109 07:21:25.813038 1916 webserver.cc:221] Webserver started. Bound to: http://0.0.0.0:8050/ I0109 07:21:25.813511 1916 tablet_server_main.cc:69] Tablet server successfully started. I0109 07:21:25.865432 2047 heartbeater.cc:292] Connected to a master server at aserbin-kudu-jepsen-master.gce.cloudera.com:7051 I0109 07:21:25.865491 2047 heartbeater.cc:359] Registering TS with master... I0109 07:21:25.865576 2047 heartbeater.cc:373] Master aserbin-kudu-jepsen-master.gce.cloudera.com:7051 requested a full tablet report, sending... I0109 07:21:25.944456 2004 raft_consensus.cc:2158] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [term 6 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 0.220s I0109 07:21:25.944537 2004 raft_consensus.cc:1744] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 591b1a1d383d4b659d7549653f4f136e in term 6. I0109 07:21:26.011034 2004 raft_consensus.cc:2202] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [term 6 FOLLOWER]: Advancing to term 7 I0109 07:21:26.011488 2004 raft_consensus.cc:1691] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate c5a3e756029444b2bf66ee0044f5bc35 for term 7 because replica has last-logged OpId of term: 6 index: 210, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 209. I0109 07:21:26.142980 2004 raft_consensus.cc:2158] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [term 7 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 0.158s I0109 07:21:26.143477 2004 raft_consensus.cc:1744] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 591b1a1d383d4b659d7549653f4f136e in term 7. I0109 07:21:26.210335 2004 raft_consensus.cc:915] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [term 7 FOLLOWER]: Refusing update from remote peer c5a3e756029444b2bf66ee0044f5bc35: Log matching property violated. Preceding OpId in replica: term: 6 index: 210. Preceding OpId from leader: term: 7 index: 210. (term mismatch) I0109 07:21:26.210392 2004 raft_consensus_state.cc:382] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e: Aborting all transactions after (but not including) 209 I0109 07:21:26.210402 2004 raft_consensus_state.cc:403] T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e: Aborting uncommitted WRITE_OP operation due to leader change: 6.210 I0109 07:21:26.211486 2047 heartbeater.cc:365] Master aserbin-kudu-jepsen-master.gce.cloudera.com:7051 was elected leader, sending a full tablet report...