I0109 07:20:10.870523 2044 tablet_server_main.cc:56] Tablet server non-default flags: --raft_heartbeat_interval_ms=50 --fs_data_dirs=/var/lib/kudu/tserver --fs_wal_dir=/var/lib/kudu/tserver --tserver_master_addrs=aserbin-kudu-jepsen-master.gce.cloudera.com:7051 --heap_profile_path=/tmp/kudu-tserver.2044 --unlock_experimental_flags=true --unlock_unsafe_flags=true --flagfile=/etc/kudu/conf/tserver.gflagfile --log_dir=/var/log/kudu --logtostderr=true Tablet server version: kudu 1.3.0-SNAPSHOT revision 3a23b058af0f4de04f8827255de405fff7bf9ee1 build type RELEASE built by jenkins at 09 Jan 2017 06:07:18 PST on aserbin-kudu-jepsen-control-node-debian8.gce.cloudera.com build id 2017-01-09_06-07-06 I0109 07:20:10.905105 2044 mem_tracker.cc:140] MemTracker: hard memory limit is 0.464404 GB I0109 07:20:10.905144 2044 mem_tracker.cc:142] MemTracker: soft memory limit is 0.278643 GB I0109 07:20:10.952708 2044 tablet_server_main.cc:63] Initializing tablet server... I0109 07:20:11.003036 2044 hybrid_clock.cc:177] HybridClock initialized. Resolution in nanos?: 1 Wait times tolerance adjustment: 1.0005 Current error: 531696 I0109 07:20:11.007120 2044 env_posix.cc:1286] Not raising process file limit of 65536; it is already as high as it can go I0109 07:20:11.007191 2044 file_cache.cc:401] Constructed file cache lbm with capacity 32768 I0109 07:20:11.008535 2044 server_base.cc:179] Could not load existing FS layout: Not found: /var/lib/kudu/tserver/instance: No such file or directory (error 2) I0109 07:20:11.008549 2044 server_base.cc:180] Creating new FS layout I0109 07:20:11.045804 2044 fs_manager.cc:362] Generated new instance metadata in path /var/lib/kudu/tserver/instance: uuid: "7897578c54854ba0a5af32eb2aa04704" format_stamp: "Formatted at 2017-01-09 15:20:11 on aserbin-kudu-jepsen-ts-3" I0109 07:20:11.053900 2044 fs_manager.cc:251] Opened local filesystem: /var/lib/kudu/tserver uuid: "7897578c54854ba0a5af32eb2aa04704" format_stamp: "Formatted at 2017-01-09 15:20:11 on aserbin-kudu-jepsen-ts-3" I0109 07:20:11.072834 2044 tablet_server_main.cc:66] Starting tablet server... I0109 07:20:11.085746 2044 rpc_server.cc:164] RPC server started. Bound to: 0.0.0.0:7050 I0109 07:20:11.085844 2044 webserver.cc:126] Starting webserver on 0.0.0.0:8050 I0109 07:20:11.085856 2044 webserver.cc:137] Document root disabled I0109 07:20:11.086041 2044 webserver.cc:221] Webserver started. Bound to: http://0.0.0.0:8050/ I0109 07:20:11.086525 2044 tablet_server_main.cc:69] Tablet server successfully started. W0109 07:20:11.142256 2171 heartbeater.cc:453] Failed to heartbeat to aserbin-kudu-jepsen-master.gce.cloudera.com:7051: Network error: Failed to ping master at aserbin-kudu-jepsen-master.gce.cloudera.com:7051: Client connection negotiation failed: client connection to 172.31.113.125:7051: connect: Connection refused (error 111) W0109 07:20:16.199254 2171 net_util.cc:124] Time spent resolving address for aserbin-kudu-jepsen-master.gce.cloudera.com: real 5.057s user 0.000s sys 0.000s I0109 07:20:16.201221 2171 heartbeater.cc:292] Connected to a master server at aserbin-kudu-jepsen-master.gce.cloudera.com:7051 I0109 07:20:16.201252 2171 heartbeater.cc:359] Registering TS with master... I0109 07:20:16.201334 2171 heartbeater.cc:373] Master aserbin-kudu-jepsen-master.gce.cloudera.com:7051 requested a full tablet report, sending... I0109 07:20:17.177718 2108 tablet_service.cc:630] Processing CreateTablet for tablet 9e99b710634a44f4a948132a5a180600 (table=rw-register-1483975178870 [id=0e0bbef280744ed1a8036e11eb0aaea0]), partition=RANGE (key) PARTITION UNBOUNDED I0109 07:20:17.184978 2108 ts_tablet_manager.cc:863] Registered tablet 9e99b710634a44f4a948132a5a180600 I0109 07:20:17.185287 2184 ts_tablet_manager.cc:725] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704: Bootstrapping tablet I0109 07:20:17.185452 2184 tablet_bootstrap.cc:381] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704: Bootstrap starting. I0109 07:20:17.186275 2184 tablet_bootstrap.cc:540] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704: Time spent opening tablet: real 0.000s user 0.000s sys 0.000s I0109 07:20:17.186357 2184 tablet_bootstrap.cc:483] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704: No blocks or log segments found. Creating new log. I0109 07:20:17.186466 2184 log.cc:351] Log is configured to *not* fsync() on all Append() calls I0109 07:20:17.186847 2184 tablet_bootstrap.cc:381] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704: No bootstrap required, opened a new log I0109 07:20:17.186872 2184 ts_tablet_manager.cc:730] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.000s I0109 07:20:17.188160 2184 raft_consensus.cc:287] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "710c236995db46699eb9055c19ac506e" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-4.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "1678276a36f849c28459de6b6788965a" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-5.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "591b1a1d383d4b659d7549653f4f136e" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-2.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "c5a3e756029444b2bf66ee0044f5bc35" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-1.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "7897578c54854ba0a5af32eb2aa04704" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-3.gce.cloudera.com" port: 7050 } } I0109 07:20:17.188187 2184 raft_consensus.cc:318] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0109 07:20:17.188195 2184 raft_consensus.cc:547] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7897578c54854ba0a5af32eb2aa04704, State: 1, Role: FOLLOWER I0109 07:20:17.188208 2184 consensus_queue.cc:175] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER I0109 07:20:17.188357 2184 ts_tablet_manager.cc:757] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s I0109 07:20:17.188388 2171 heartbeater.cc:365] Master aserbin-kudu-jepsen-master.gce.cloudera.com:7051 was elected leader, sending a full tablet report... I0109 07:20:17.222173 2187 raft_consensus.cc:410] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0109 07:20:17.222250 2187 raft_consensus.cc:2158] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [term 0 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 0.170s I0109 07:20:17.222312 2187 raft_consensus.cc:434] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "710c236995db46699eb9055c19ac506e" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-4.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "1678276a36f849c28459de6b6788965a" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-5.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "591b1a1d383d4b659d7549653f4f136e" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-2.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "c5a3e756029444b2bf66ee0044f5bc35" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-1.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "7897578c54854ba0a5af32eb2aa04704" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-3.gce.cloudera.com" port: 7050 } } I0109 07:20:17.419868 2127 raft_consensus.cc:1719] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [term 0 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c5a3e756029444b2bf66ee0044f5bc35 for term 1 because replica is already servicing an update from a current leader or another vote. I0109 07:20:17.420083 2128 raft_consensus.cc:2158] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [term 0 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 0.188s I0109 07:20:17.420104 2128 raft_consensus.cc:1744] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 591b1a1d383d4b659d7549653f4f136e in term 0. W0109 07:20:17.420147 2128 rpcz_store.cc:234] Call kudu.consensus.ConsensusService.RequestConsensusVote from 172.31.113.2:40097 (request call id 1) took 161ms (client timeout 197). W0109 07:20:17.420205 2128 rpcz_store.cc:238] Trace: 0109 07:20:17.258330 (+ 0us) service_pool.cc:143] Inserting onto call queue 0109 07:20:17.258342 (+ 12us) service_pool.cc:202] Handling call 0109 07:20:17.420125 (+161783us) spinlock_profiling.cc:255] Waited 162 ms on lock 0x45c8f40. stack: 0109 07:20:17.420134 (+ 9us) inbound_call.cc:130] Queueing success response Metrics: {"spinlock_wait_cycles":404126464} I0109 07:20:17.420338 2187 leader_election.cc:215] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 710c236995db46699eb9055c19ac506e I0109 07:20:17.420578 2187 leader_election.cc:215] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 1678276a36f849c28459de6b6788965a I0109 07:20:17.420691 2187 leader_election.cc:215] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 591b1a1d383d4b659d7549653f4f136e I0109 07:20:17.420888 2187 leader_election.cc:215] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer c5a3e756029444b2bf66ee0044f5bc35 I0109 07:20:17.422674 2063 leader_election.cc:347] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [CANDIDATE]: Term 1 pre-election: Vote granted by peer 1678276a36f849c28459de6b6788965a I0109 07:20:17.423182 2062 leader_election.cc:347] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [CANDIDATE]: Term 1 pre-election: Vote granted by peer c5a3e756029444b2bf66ee0044f5bc35 I0109 07:20:17.423200 2062 leader_election.cc:242] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. I0109 07:20:17.423347 2192 raft_consensus.cc:2158] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [term 0 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 0.172s I0109 07:20:17.423367 2192 raft_consensus.cc:1968] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [term 0 FOLLOWER]: Leader pre-election won for term 1 I0109 07:20:17.423373 2192 raft_consensus.cc:410] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0109 07:20:17.423388 2192 raft_consensus.cc:2158] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [term 0 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 0.153s I0109 07:20:17.423409 2192 raft_consensus.cc:2202] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [term 0 FOLLOWER]: Advancing to term 1 I0109 07:20:17.423745 2192 raft_consensus.cc:434] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "710c236995db46699eb9055c19ac506e" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-4.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "1678276a36f849c28459de6b6788965a" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-5.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "591b1a1d383d4b659d7549653f4f136e" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-2.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "c5a3e756029444b2bf66ee0044f5bc35" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-1.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "7897578c54854ba0a5af32eb2aa04704" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-3.gce.cloudera.com" port: 7050 } } I0109 07:20:17.457677 2065 leader_election.cc:361] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [CANDIDATE]: Term 1 pre-election: Vote denied by peer 591b1a1d383d4b659d7549653f4f136e. Message: Service unavailable: T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7897578c54854ba0a5af32eb2aa04704 for term 1 because replica is already servicing an update from a current leader or another vote. I0109 07:20:17.458766 2064 leader_election.cc:361] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [CANDIDATE]: Term 1 pre-election: Vote denied by peer 710c236995db46699eb9055c19ac506e. Message: Service unavailable: T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7897578c54854ba0a5af32eb2aa04704 for term 1 because replica is already servicing an update from a current leader or another vote. I0109 07:20:17.621485 2127 raft_consensus.cc:1674] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 710c236995db46699eb9055c19ac506e in current term 1: Already voted for candidate 7897578c54854ba0a5af32eb2aa04704 in this term. W0109 07:20:17.621634 2127 rpcz_store.cc:234] Call kudu.consensus.ConsensusService.RequestConsensusVote from 172.31.114.38:41916 (request call id 2) took 164ms (client timeout 208). W0109 07:20:17.621712 2127 rpcz_store.cc:238] Trace: 0109 07:20:17.456669 (+ 0us) service_pool.cc:143] Inserting onto call queue 0109 07:20:17.456690 (+ 21us) service_pool.cc:202] Handling call 0109 07:20:17.621604 (+164914us) spinlock_profiling.cc:255] Waited 165 ms on lock 0x45c8f40. stack: 0000000001a20b21 00000000009d8b77 00000000007e48af 0000000000a8971f 0000000000a8a342 0000000001a2dc19 00007faf4757c0a3 00007faf45e0c87c ffffffffffffffff 0109 07:20:17.621617 (+ 13us) inbound_call.cc:130] Queueing success response Metrics: {"spinlock_wait_cycles":411870464} I0109 07:20:17.621763 2128 raft_consensus.cc:1719] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 591b1a1d383d4b659d7549653f4f136e for term 1 because replica is already servicing an update from a current leader or another vote. W0109 07:20:17.621816 2128 rpcz_store.cc:234] Call kudu.consensus.ConsensusService.RequestConsensusVote from 172.31.113.2:40097 (request call id 2) took 163ms (client timeout 196). W0109 07:20:17.621845 2128 rpcz_store.cc:238] Trace: 0109 07:20:17.457874 (+ 0us) service_pool.cc:143] Inserting onto call queue 0109 07:20:17.457884 (+ 10us) service_pool.cc:202] Handling call 0109 07:20:17.621801 (+163917us) spinlock_profiling.cc:255] Waited 164 ms on lock 0x45c8f40. stack: 0000000001a20b21 00000000009d8c7a 00000000007e48af 0000000000a8971f 0000000000a8a342 0000000001a2dc19 00007faf4757c0a3 00007faf45e0c87c ffffffffffffffff 0109 07:20:17.621807 (+ 6us) inbound_call.cc:130] Queueing success response Metrics: {"spinlock_wait_cycles":409648384} I0109 07:20:17.621996 2192 leader_election.cc:215] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [CANDIDATE]: Term 1 election: Requesting vote from peer 710c236995db46699eb9055c19ac506e I0109 07:20:17.622061 2192 leader_election.cc:215] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [CANDIDATE]: Term 1 election: Requesting vote from peer 1678276a36f849c28459de6b6788965a I0109 07:20:17.622104 2192 leader_election.cc:215] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [CANDIDATE]: Term 1 election: Requesting vote from peer 591b1a1d383d4b659d7549653f4f136e I0109 07:20:17.622153 2192 leader_election.cc:215] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [CANDIDATE]: Term 1 election: Requesting vote from peer c5a3e756029444b2bf66ee0044f5bc35 I0109 07:20:17.622656 2063 leader_election.cc:361] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [CANDIDATE]: Term 1 election: Vote denied by peer 1678276a36f849c28459de6b6788965a. Message: Invalid argument: T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 7897578c54854ba0a5af32eb2aa04704 in current term 1: Already voted for candidate 710c236995db46699eb9055c19ac506e in this term. I0109 07:20:17.622890 2062 leader_election.cc:361] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [CANDIDATE]: Term 1 election: Vote denied by peer c5a3e756029444b2bf66ee0044f5bc35. Message: Invalid argument: T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 7897578c54854ba0a5af32eb2aa04704 in current term 1: Already voted for candidate 710c236995db46699eb9055c19ac506e in this term. I0109 07:20:17.658215 2064 leader_election.cc:361] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [CANDIDATE]: Term 1 election: Vote denied by peer 710c236995db46699eb9055c19ac506e. Message: Service unavailable: T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [term 1 LEADER]: Leader election vote request: Denying vote to candidate 7897578c54854ba0a5af32eb2aa04704 for term 1 because replica is already servicing an update from a current leader or another vote. I0109 07:20:17.658270 2064 leader_election.cc:242] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. I0109 07:20:17.658340 2192 raft_consensus.cc:2158] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 0.222s I0109 07:20:17.658354 2192 raft_consensus.cc:1905] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority I0109 07:20:17.662119 2128 raft_consensus.cc:915] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [term 1 FOLLOWER]: Refusing update from remote peer 710c236995db46699eb9055c19ac506e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch) W0109 07:20:17.775588 2065 leader_election.cc:271] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 591b1a1d383d4b659d7549653f4f136e: Timed out: RequestConsensusVote RPC to 172.31.113.2:7050 timed out after 0.153s (SENT) I0109 07:20:17.805172 2128 raft_consensus.cc:1705] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 591b1a1d383d4b659d7549653f4f136e for term 2 because replica is either leader or believes a valid leader to be alive. I0109 07:20:30.365918 2209 tablet_server_main.cc:56] Tablet server non-default flags: --raft_heartbeat_interval_ms=50 --fs_data_dirs=/var/lib/kudu/tserver --fs_wal_dir=/var/lib/kudu/tserver --tserver_master_addrs=aserbin-kudu-jepsen-master.gce.cloudera.com:7051 --heap_profile_path=/tmp/kudu-tserver.2209 --unlock_experimental_flags=true --unlock_unsafe_flags=true --flagfile=/etc/kudu/conf/tserver.gflagfile --log_dir=/var/log/kudu --logtostderr=true Tablet server version: kudu 1.3.0-SNAPSHOT revision 3a23b058af0f4de04f8827255de405fff7bf9ee1 build type RELEASE built by jenkins at 09 Jan 2017 06:07:18 PST on aserbin-kudu-jepsen-control-node-debian8.gce.cloudera.com build id 2017-01-09_06-07-06 I0109 07:20:30.366065 2209 mem_tracker.cc:140] MemTracker: hard memory limit is 0.464404 GB I0109 07:20:30.366075 2209 mem_tracker.cc:142] MemTracker: soft memory limit is 0.278643 GB I0109 07:20:30.367205 2209 tablet_server_main.cc:63] Initializing tablet server... I0109 07:20:30.416787 2209 hybrid_clock.cc:177] HybridClock initialized. Resolution in nanos?: 1 Wait times tolerance adjustment: 1.0005 Current error: 541196 I0109 07:20:30.417577 2209 env_posix.cc:1286] Not raising process file limit of 65536; it is already as high as it can go I0109 07:20:30.417605 2209 file_cache.cc:401] Constructed file cache lbm with capacity 32768 I0109 07:20:30.418586 2209 fs_manager.cc:251] Opened local filesystem: /var/lib/kudu/tserver uuid: "7897578c54854ba0a5af32eb2aa04704" format_stamp: "Formatted at 2017-01-09 15:20:11 on aserbin-kudu-jepsen-ts-3" I0109 07:20:30.421910 2209 ts_tablet_manager.cc:701] Loading metadata for tablet 9e99b710634a44f4a948132a5a180600 I0109 07:20:30.422206 2209 ts_tablet_manager.cc:863] Registered tablet 9e99b710634a44f4a948132a5a180600 I0109 07:20:30.422372 2223 ts_tablet_manager.cc:725] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704: Bootstrapping tablet I0109 07:20:30.422544 2223 tablet_bootstrap.cc:381] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704: Bootstrap starting. I0109 07:20:30.422762 2223 tablet_bootstrap.cc:540] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704: Time spent opening tablet: real 0.000s user 0.000s sys 0.000s I0109 07:20:30.422839 2223 tablet_bootstrap.cc:596] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704: Will attempt to recover log segment /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600/wal-000000001 to /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery/wal-000000001 I0109 07:20:30.422852 2223 tablet_bootstrap.cc:604] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704: Moving log directory /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600 to recovery directory /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery in preparation for log replay I0109 07:20:30.423044 2223 log_util.cc:315] Log segment /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery/wal-000000001 has no footer. This segment was likely being written when the server previously shut down. I0109 07:20:30.423056 2223 log_reader.cc:160] Log segment /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery/wal-000000001 was likely left in-progress after a previous crash. Will try to rebuild footer by scanning data. I0109 07:20:30.424685 2209 tablet_server_main.cc:66] Starting tablet server... I0109 07:20:30.425588 2223 log_util.cc:569] Scanning /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery/wal-000000001 for valid entry headers following offset 10988... I0109 07:20:30.497942 2223 log_util.cc:606] Found no log entry headers I0109 07:20:30.498004 2223 log_util.cc:218] Ignoring log segment corruption in /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery/wal-000000001 because there are no log entries following the corrupted one. The server probably crashed in the middle of writing an entry to the write-ahead log or downloaded an active log via tablet copy. Error detail: Corruption: CRC mismatch in log entry header: Log file corruption detected. Failed trying to read batch #0 at offset 10976 for log segment /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery/wal-000000001: Prior entries: [off=10711 REPLICATE (1.47)] [off=10745 COMMIT (1.47)] [off=10942 REPLICATE (1.48)] [off=10976 COMMIT (1.48)] I0109 07:20:30.498029 2223 log_util.cc:368] Successfully rebuilt footer for segment: /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery/wal-000000001 (valid entries through byte offset 10976) I0109 07:20:30.498095 2223 tablet.cc:983] T 9e99b710634a44f4a948132a5a180600 Rewinding schema during bootstrap to Schema [ 0:key[string NOT NULL], 1:value[int32 NOT NULL] ] I0109 07:20:30.498323 2223 log.cc:351] Log is configured to *not* fsync() on all Append() calls I0109 07:20:30.502234 2223 tablet_bootstrap.cc:381] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704: Bootstrap replayed 1/1 log segments. Stats: ops{read=48 overwritten=0 applied=48 ignored=0} inserts{seen=47 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I0109 07:20:30.502282 2223 tablet_bootstrap.cc:1017] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704: ReplayState: Previous OpId: 1.48, Committed OpId: 1.48, Pending Replicates: 0, Pending Commits: 0 I0109 07:20:30.502723 2223 tablet_bootstrap.cc:633] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704: Preparing to delete log recovery files and directory /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery I0109 07:20:30.502779 2223 tablet_bootstrap.cc:636] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704: Renaming log recovery dir from /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery to /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery-1483975230502775 I0109 07:20:30.502841 2223 tablet_bootstrap.cc:646] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704: Deleting all files from renamed log recovery directory /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery-1483975230502775 I0109 07:20:30.503072 2223 tablet_bootstrap.cc:649] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704: Completed deletion of old log recovery files and directory /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery-1483975230502775 I0109 07:20:30.503110 2223 tablet_bootstrap.cc:381] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704: Bootstrap complete. I0109 07:20:30.510079 2223 ts_tablet_manager.cc:730] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704: Time spent bootstrapping tablet: real 0.088s user 0.020s sys 0.024s I0109 07:20:30.510628 2223 raft_consensus.cc:287] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "710c236995db46699eb9055c19ac506e" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-4.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "1678276a36f849c28459de6b6788965a" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-5.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "591b1a1d383d4b659d7549653f4f136e" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-2.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "c5a3e756029444b2bf66ee0044f5bc35" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-1.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "7897578c54854ba0a5af32eb2aa04704" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-3.gce.cloudera.com" port: 7050 } } I0109 07:20:30.510665 2223 raft_consensus.cc:547] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7897578c54854ba0a5af32eb2aa04704, State: 1, Role: FOLLOWER I0109 07:20:30.510677 2223 consensus_queue.cc:175] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 48, Last appended: 1.48, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER I0109 07:20:30.510849 2223 ts_tablet_manager.cc:757] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s I0109 07:20:30.518244 2209 rpc_server.cc:164] RPC server started. Bound to: 0.0.0.0:7050 I0109 07:20:30.518317 2209 webserver.cc:126] Starting webserver on 0.0.0.0:8050 I0109 07:20:30.518323 2209 webserver.cc:137] Document root disabled I0109 07:20:30.518470 2209 webserver.cc:221] Webserver started. Bound to: http://0.0.0.0:8050/ I0109 07:20:30.518849 2209 tablet_server_main.cc:69] Tablet server successfully started. I0109 07:20:30.570869 2339 heartbeater.cc:292] Connected to a master server at aserbin-kudu-jepsen-master.gce.cloudera.com:7051 I0109 07:20:30.570915 2339 heartbeater.cc:359] Registering TS with master... I0109 07:20:30.570984 2339 heartbeater.cc:373] Master aserbin-kudu-jepsen-master.gce.cloudera.com:7051 requested a full tablet report, sending... I0109 07:20:30.668668 2296 raft_consensus.cc:2158] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 0.177s I0109 07:20:30.668715 2296 raft_consensus.cc:1744] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 710c236995db46699eb9055c19ac506e in term 1. I0109 07:20:31.006983 2261 raft_consensus.cc:410] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0109 07:20:31.007060 2261 raft_consensus.cc:2158] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 0.158s I0109 07:20:31.007200 2261 raft_consensus.cc:434] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "710c236995db46699eb9055c19ac506e" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-4.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "1678276a36f849c28459de6b6788965a" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-5.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "591b1a1d383d4b659d7549653f4f136e" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-2.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "c5a3e756029444b2bf66ee0044f5bc35" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-1.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "7897578c54854ba0a5af32eb2aa04704" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-3.gce.cloudera.com" port: 7050 } } I0109 07:20:31.205550 2261 leader_election.cc:215] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 710c236995db46699eb9055c19ac506e I0109 07:20:31.205601 2261 leader_election.cc:215] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 1678276a36f849c28459de6b6788965a I0109 07:20:31.205616 2261 leader_election.cc:215] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 591b1a1d383d4b659d7549653f4f136e I0109 07:20:31.205642 2261 leader_election.cc:215] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer c5a3e756029444b2bf66ee0044f5bc35 I0109 07:20:31.206014 2296 raft_consensus.cc:1691] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 591b1a1d383d4b659d7549653f4f136e for term 2 because replica has last-logged OpId of term: 1 index: 48, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 46. W0109 07:20:31.207319 2219 leader_election.cc:271] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1678276a36f849c28459de6b6788965a: Network error: Client connection negotiation failed: client connection to 172.31.114.39:7050: connect: Connection refused (error 111) I0109 07:20:31.208328 2221 leader_election.cc:347] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [CANDIDATE]: Term 2 pre-election: Vote granted by peer 591b1a1d383d4b659d7549653f4f136e I0109 07:20:31.208686 2218 leader_election.cc:361] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [CANDIDATE]: Term 2 pre-election: Vote denied by peer c5a3e756029444b2bf66ee0044f5bc35. Message: Invalid argument: T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7897578c54854ba0a5af32eb2aa04704 for term 2 because replica has last-logged OpId of term: 1 index: 49, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 48. I0109 07:20:31.275632 2220 leader_election.cc:361] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [CANDIDATE]: Term 2 pre-election: Vote denied by peer 710c236995db46699eb9055c19ac506e. Message: Service unavailable: T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7897578c54854ba0a5af32eb2aa04704 for term 2 because replica is already servicing an update from a current leader or another vote. I0109 07:20:31.275696 2220 leader_election.cc:242] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. I0109 07:20:31.275902 2353 raft_consensus.cc:2158] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 0.161s I0109 07:20:31.275918 2353 raft_consensus.cc:1905] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority I0109 07:20:31.276376 2296 raft_consensus.cc:2158] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 0.290s I0109 07:20:31.276391 2296 raft_consensus.cc:1744] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 710c236995db46699eb9055c19ac506e in term 1. I0109 07:20:31.477047 2296 raft_consensus.cc:2202] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [term 1 FOLLOWER]: Advancing to term 2 I0109 07:20:31.477161 2296 raft_consensus.cc:2158] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [term 2 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 0.332s I0109 07:20:31.477627 2296 raft_consensus.cc:1744] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 710c236995db46699eb9055c19ac506e in term 2. I0109 07:20:31.572273 2339 heartbeater.cc:365] Master aserbin-kudu-jepsen-master.gce.cloudera.com:7051 was elected leader, sending a full tablet report... I0109 07:20:31.976325 2261 raft_consensus.cc:410] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0109 07:20:31.976395 2261 raft_consensus.cc:2158] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [term 2 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 0.303s I0109 07:20:31.976457 2261 raft_consensus.cc:434] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "710c236995db46699eb9055c19ac506e" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-4.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "1678276a36f849c28459de6b6788965a" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-5.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "591b1a1d383d4b659d7549653f4f136e" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-2.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "c5a3e756029444b2bf66ee0044f5bc35" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-1.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "7897578c54854ba0a5af32eb2aa04704" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-3.gce.cloudera.com" port: 7050 } } I0109 07:20:32.172302 2261 leader_election.cc:215] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [CANDIDATE]: Term 3 pre-election: Requesting pre-vote from peer 710c236995db46699eb9055c19ac506e I0109 07:20:32.172369 2261 leader_election.cc:215] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [CANDIDATE]: Term 3 pre-election: Requesting pre-vote from peer 1678276a36f849c28459de6b6788965a I0109 07:20:32.172382 2261 leader_election.cc:215] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [CANDIDATE]: Term 3 pre-election: Requesting pre-vote from peer 591b1a1d383d4b659d7549653f4f136e I0109 07:20:32.172396 2261 leader_election.cc:215] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [CANDIDATE]: Term 3 pre-election: Requesting pre-vote from peer c5a3e756029444b2bf66ee0044f5bc35 I0109 07:20:32.172685 2294 raft_consensus.cc:1719] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c5a3e756029444b2bf66ee0044f5bc35 for term 3 because replica is already servicing an update from a current leader or another vote. I0109 07:20:32.172880 2295 raft_consensus.cc:1719] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1678276a36f849c28459de6b6788965a for term 3 because replica is already servicing an update from a current leader or another vote. I0109 07:20:32.172984 2296 raft_consensus.cc:1691] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 591b1a1d383d4b659d7549653f4f136e for term 3 because replica has last-logged OpId of term: 1 index: 48, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 46. I0109 07:20:32.173137 2221 leader_election.cc:347] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [CANDIDATE]: Term 3 pre-election: Vote granted by peer 591b1a1d383d4b659d7549653f4f136e I0109 07:20:32.174415 2219 leader_election.cc:361] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [CANDIDATE]: Term 3 pre-election: Vote denied by peer 1678276a36f849c28459de6b6788965a. Message: Invalid argument: T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7897578c54854ba0a5af32eb2aa04704 for term 3 because replica has last-logged OpId of term: 1 index: 49, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 48. I0109 07:20:32.453196 2261 raft_consensus.cc:410] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0109 07:20:32.453268 2261 raft_consensus.cc:2158] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [term 2 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 0.320s I0109 07:20:32.453325 2261 raft_consensus.cc:434] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "710c236995db46699eb9055c19ac506e" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-4.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "1678276a36f849c28459de6b6788965a" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-5.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "591b1a1d383d4b659d7549653f4f136e" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-2.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "c5a3e756029444b2bf66ee0044f5bc35" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-1.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "7897578c54854ba0a5af32eb2aa04704" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-3.gce.cloudera.com" port: 7050 } } W0109 07:20:32.475890 2218 leader_election.cc:271] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c5a3e756029444b2bf66ee0044f5bc35: Timed out: RequestConsensusVote RPC to 172.31.112.255:7050 timed out after 0.303s (SENT) W0109 07:20:32.476191 2220 leader_election.cc:271] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 710c236995db46699eb9055c19ac506e: Timed out: RequestConsensusVote RPC to 172.31.114.38:7050 timed out after 0.303s (SENT) I0109 07:20:32.476204 2220 leader_election.cc:242] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. I0109 07:20:32.649698 2261 leader_election.cc:215] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [CANDIDATE]: Term 3 pre-election: Requesting pre-vote from peer 710c236995db46699eb9055c19ac506e I0109 07:20:32.649770 2261 leader_election.cc:215] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [CANDIDATE]: Term 3 pre-election: Requesting pre-vote from peer 1678276a36f849c28459de6b6788965a I0109 07:20:32.649785 2261 leader_election.cc:215] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [CANDIDATE]: Term 3 pre-election: Requesting pre-vote from peer 591b1a1d383d4b659d7549653f4f136e I0109 07:20:32.649801 2261 leader_election.cc:215] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [CANDIDATE]: Term 3 pre-election: Requesting pre-vote from peer c5a3e756029444b2bf66ee0044f5bc35 I0109 07:20:32.650040 2355 raft_consensus.cc:2158] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [term 2 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 0.252s I0109 07:20:32.650056 2355 raft_consensus.cc:1905] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority I0109 07:20:32.650074 2296 raft_consensus.cc:2202] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [term 2 FOLLOWER]: Advancing to term 3 I0109 07:20:32.650110 2296 raft_consensus.cc:2158] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [term 3 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 0.476s I0109 07:20:32.650473 2296 raft_consensus.cc:1744] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5a3e756029444b2bf66ee0044f5bc35 in term 3. I0109 07:20:32.825100 2221 leader_election.cc:361] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [CANDIDATE]: Term 3 pre-election: Vote denied by peer 591b1a1d383d4b659d7549653f4f136e. Message: Service unavailable: T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7897578c54854ba0a5af32eb2aa04704 for term 3 because replica is already servicing an update from a current leader or another vote. I0109 07:20:32.825312 2296 raft_consensus.cc:1674] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 591b1a1d383d4b659d7549653f4f136e in current term 3: Already voted for candidate c5a3e756029444b2bf66ee0044f5bc35 in this term. I0109 07:20:32.825541 2219 leader_election.cc:361] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [CANDIDATE]: Term 3 pre-election: Vote denied by peer 1678276a36f849c28459de6b6788965a. Message: Service unavailable: T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7897578c54854ba0a5af32eb2aa04704 for term 3 because replica is already servicing an update from a current leader or another vote. I0109 07:20:32.825922 2296 raft_consensus.cc:1674] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1678276a36f849c28459de6b6788965a in current term 3: Already voted for candidate c5a3e756029444b2bf66ee0044f5bc35 in this term. I0109 07:20:32.836185 2218 leader_election.cc:361] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [CANDIDATE]: Term 3 pre-election: Vote denied by peer c5a3e756029444b2bf66ee0044f5bc35. Message: Invalid argument: T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7897578c54854ba0a5af32eb2aa04704 in current term 3: Already voted for candidate c5a3e756029444b2bf66ee0044f5bc35 in this term. I0109 07:20:32.836241 2218 leader_election.cc:242] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. I0109 07:20:32.836345 2296 raft_consensus.cc:2158] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [term 3 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 0.282s I0109 07:20:32.836360 2296 raft_consensus.cc:1744] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5a3e756029444b2bf66ee0044f5bc35 in term 3. I0109 07:20:32.836587 2356 raft_consensus.cc:2158] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [term 3 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 0.466s I0109 07:20:32.836603 2356 raft_consensus.cc:1905] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority W0109 07:20:32.970664 2220 leader_election.cc:271] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 710c236995db46699eb9055c19ac506e: Timed out: RequestConsensusVote RPC to 172.31.114.38:7050 timed out after 0.320s (SENT) I0109 07:20:33.456398 2261 raft_consensus.cc:410] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0109 07:20:33.456490 2261 raft_consensus.cc:2158] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [term 3 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 0.378s I0109 07:20:33.456548 2261 raft_consensus.cc:434] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "710c236995db46699eb9055c19ac506e" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-4.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "1678276a36f849c28459de6b6788965a" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-5.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "591b1a1d383d4b659d7549653f4f136e" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-2.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "c5a3e756029444b2bf66ee0044f5bc35" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-1.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "7897578c54854ba0a5af32eb2aa04704" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-3.gce.cloudera.com" port: 7050 } } I0109 07:20:33.653440 2296 raft_consensus.cc:2158] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [term 3 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 0.241s I0109 07:20:33.653496 2296 raft_consensus.cc:1744] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1678276a36f849c28459de6b6788965a in term 3. I0109 07:20:33.653671 2261 leader_election.cc:215] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [CANDIDATE]: Term 4 pre-election: Requesting pre-vote from peer 710c236995db46699eb9055c19ac506e I0109 07:20:33.653753 2261 leader_election.cc:215] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [CANDIDATE]: Term 4 pre-election: Requesting pre-vote from peer 1678276a36f849c28459de6b6788965a I0109 07:20:33.653815 2261 leader_election.cc:215] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [CANDIDATE]: Term 4 pre-election: Requesting pre-vote from peer 591b1a1d383d4b659d7549653f4f136e I0109 07:20:33.653869 2261 leader_election.cc:215] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [CANDIDATE]: Term 4 pre-election: Requesting pre-vote from peer c5a3e756029444b2bf66ee0044f5bc35 I0109 07:20:33.654340 2219 leader_election.cc:361] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [CANDIDATE]: Term 4 pre-election: Vote denied by peer 1678276a36f849c28459de6b6788965a. Message: Invalid argument: T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7897578c54854ba0a5af32eb2aa04704 for term 4 because replica has last-logged OpId of term: 1 index: 49, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 48. I0109 07:20:33.666379 2221 leader_election.cc:361] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [CANDIDATE]: Term 4 pre-election: Vote denied by peer 591b1a1d383d4b659d7549653f4f136e. Message: Service unavailable: T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7897578c54854ba0a5af32eb2aa04704 for term 4 because replica is already servicing an update from a current leader or another vote. I0109 07:20:33.666564 2296 raft_consensus.cc:1691] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 591b1a1d383d4b659d7549653f4f136e for term 4 because replica has last-logged OpId of term: 1 index: 48, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 46. I0109 07:20:33.865031 2296 raft_consensus.cc:2202] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [term 3 FOLLOWER]: Advancing to term 4 I0109 07:20:33.865124 2296 raft_consensus.cc:2158] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [term 4 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 0.217s I0109 07:20:33.865484 2296 raft_consensus.cc:1744] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1678276a36f849c28459de6b6788965a in term 4. W0109 07:20:34.032656 2218 leader_election.cc:271] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer c5a3e756029444b2bf66ee0044f5bc35: Timed out: RequestConsensusVote RPC to 172.31.112.255:7050 timed out after 0.378s (SENT) I0109 07:20:34.032709 2218 leader_election.cc:242] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. W0109 07:20:34.032783 2220 leader_election.cc:271] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 710c236995db46699eb9055c19ac506e: Timed out: RequestConsensusVote RPC to 172.31.114.38:7050 timed out after 0.378s (SENT) I0109 07:20:34.032956 2358 raft_consensus.cc:2158] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [term 4 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 0.405s I0109 07:20:34.032973 2358 raft_consensus.cc:1905] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority I0109 07:20:34.156345 2296 raft_consensus.cc:915] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [term 4 FOLLOWER]: Refusing update from remote peer 1678276a36f849c28459de6b6788965a: Log matching property violated. Preceding OpId in replica: term: 1 index: 48. Preceding OpId from leader: term: 4 index: 50. (index mismatch) I0109 07:20:37.456079 2261 raft_consensus.cc:410] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1678276a36f849c28459de6b6788965a) I0109 07:20:37.456167 2261 raft_consensus.cc:2158] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [term 4 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 0.197s I0109 07:20:37.456233 2261 raft_consensus.cc:434] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "710c236995db46699eb9055c19ac506e" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-4.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "1678276a36f849c28459de6b6788965a" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-5.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "591b1a1d383d4b659d7549653f4f136e" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-2.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "c5a3e756029444b2bf66ee0044f5bc35" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-1.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "7897578c54854ba0a5af32eb2aa04704" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-3.gce.cloudera.com" port: 7050 } } I0109 07:20:37.655530 2261 leader_election.cc:215] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [CANDIDATE]: Term 5 pre-election: Requesting pre-vote from peer 710c236995db46699eb9055c19ac506e I0109 07:20:37.655599 2261 leader_election.cc:215] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [CANDIDATE]: Term 5 pre-election: Requesting pre-vote from peer 1678276a36f849c28459de6b6788965a I0109 07:20:37.655634 2261 leader_election.cc:215] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [CANDIDATE]: Term 5 pre-election: Requesting pre-vote from peer 591b1a1d383d4b659d7549653f4f136e I0109 07:20:37.655647 2261 leader_election.cc:215] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [CANDIDATE]: Term 5 pre-election: Requesting pre-vote from peer c5a3e756029444b2bf66ee0044f5bc35 W0109 07:20:37.656365 2219 leader_election.cc:271] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 1678276a36f849c28459de6b6788965a: Network error: Client connection negotiation failed: client connection to 172.31.114.39:7050: connect: Connection refused (error 111) W0109 07:20:37.656424 2221 leader_election.cc:271] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 591b1a1d383d4b659d7549653f4f136e: Network error: Client connection negotiation failed: client connection to 172.31.113.2:7050: connect: Connection refused (error 111) I0109 07:20:37.657759 2220 leader_election.cc:347] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [CANDIDATE]: Term 5 pre-election: Vote granted by peer 710c236995db46699eb9055c19ac506e I0109 07:20:37.658165 2296 raft_consensus.cc:2158] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [term 4 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 0.188s I0109 07:20:37.658192 2296 raft_consensus.cc:1744] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 710c236995db46699eb9055c19ac506e in term 4. I0109 07:20:44.547161 2383 tablet_server_main.cc:56] Tablet server non-default flags: --raft_heartbeat_interval_ms=50 --fs_data_dirs=/var/lib/kudu/tserver --fs_wal_dir=/var/lib/kudu/tserver --tserver_master_addrs=aserbin-kudu-jepsen-master.gce.cloudera.com:7051 --heap_profile_path=/tmp/kudu-tserver.2383 --unlock_experimental_flags=true --unlock_unsafe_flags=true --flagfile=/etc/kudu/conf/tserver.gflagfile --log_dir=/var/log/kudu --logtostderr=true Tablet server version: kudu 1.3.0-SNAPSHOT revision 3a23b058af0f4de04f8827255de405fff7bf9ee1 build type RELEASE built by jenkins at 09 Jan 2017 06:07:18 PST on aserbin-kudu-jepsen-control-node-debian8.gce.cloudera.com build id 2017-01-09_06-07-06 I0109 07:20:44.547327 2383 mem_tracker.cc:140] MemTracker: hard memory limit is 0.464404 GB I0109 07:20:44.547343 2383 mem_tracker.cc:142] MemTracker: soft memory limit is 0.278643 GB I0109 07:20:44.548578 2383 tablet_server_main.cc:63] Initializing tablet server... I0109 07:20:44.597954 2383 hybrid_clock.cc:177] HybridClock initialized. Resolution in nanos?: 1 Wait times tolerance adjustment: 1.0005 Current error: 548196 I0109 07:20:44.598726 2383 env_posix.cc:1286] Not raising process file limit of 65536; it is already as high as it can go I0109 07:20:44.598763 2383 file_cache.cc:401] Constructed file cache lbm with capacity 32768 I0109 07:20:44.599933 2383 fs_manager.cc:251] Opened local filesystem: /var/lib/kudu/tserver uuid: "7897578c54854ba0a5af32eb2aa04704" format_stamp: "Formatted at 2017-01-09 15:20:11 on aserbin-kudu-jepsen-ts-3" I0109 07:20:44.604136 2383 ts_tablet_manager.cc:701] Loading metadata for tablet 9e99b710634a44f4a948132a5a180600 I0109 07:20:44.604399 2383 ts_tablet_manager.cc:863] Registered tablet 9e99b710634a44f4a948132a5a180600 I0109 07:20:44.604573 2397 ts_tablet_manager.cc:725] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704: Bootstrapping tablet I0109 07:20:44.604753 2397 tablet_bootstrap.cc:381] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704: Bootstrap starting. I0109 07:20:44.604989 2397 tablet_bootstrap.cc:540] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704: Time spent opening tablet: real 0.000s user 0.000s sys 0.000s I0109 07:20:44.605082 2397 tablet_bootstrap.cc:596] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704: Will attempt to recover log segment /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600/wal-000000001 to /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery/wal-000000001 I0109 07:20:44.605108 2397 tablet_bootstrap.cc:604] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704: Moving log directory /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600 to recovery directory /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery in preparation for log replay I0109 07:20:44.605294 2397 log_util.cc:315] Log segment /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery/wal-000000001 has no footer. This segment was likely being written when the server previously shut down. I0109 07:20:44.605304 2397 log_reader.cc:160] Log segment /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery/wal-000000001 was likely left in-progress after a previous crash. Will try to rebuild footer by scanning data. I0109 07:20:44.606861 2383 tablet_server_main.cc:66] Starting tablet server... I0109 07:20:44.608099 2397 log_util.cc:569] Scanning /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery/wal-000000001 for valid entry headers following offset 13812... I0109 07:20:44.654261 2397 log_util.cc:606] Found no log entry headers I0109 07:20:44.654321 2397 log_util.cc:218] Ignoring log segment corruption in /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery/wal-000000001 because there are no log entries following the corrupted one. The server probably crashed in the middle of writing an entry to the write-ahead log or downloaded an active log via tablet copy. Error detail: Corruption: CRC mismatch in log entry header: Log file corruption detected. Failed trying to read batch #0 at offset 13800 for log segment /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery/wal-000000001: Prior entries: [off=13535 REPLICATE (4.60)] [off=13569 COMMIT (4.60)] [off=13766 REPLICATE (4.61)] [off=13800 COMMIT (4.61)] I0109 07:20:44.654331 2397 log_util.cc:368] Successfully rebuilt footer for segment: /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery/wal-000000001 (valid entries through byte offset 13800) I0109 07:20:44.654394 2397 tablet.cc:983] T 9e99b710634a44f4a948132a5a180600 Rewinding schema during bootstrap to Schema [ 0:key[string NOT NULL], 1:value[int32 NOT NULL] ] I0109 07:20:44.654605 2397 log.cc:351] Log is configured to *not* fsync() on all Append() calls I0109 07:20:44.657508 2397 tablet_bootstrap.cc:381] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704: Bootstrap replayed 1/1 log segments. Stats: ops{read=61 overwritten=0 applied=61 ignored=0} inserts{seen=59 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I0109 07:20:44.657531 2397 tablet_bootstrap.cc:1017] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704: ReplayState: Previous OpId: 4.61, Committed OpId: 4.61, Pending Replicates: 0, Pending Commits: 0 I0109 07:20:44.657882 2397 tablet_bootstrap.cc:633] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704: Preparing to delete log recovery files and directory /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery I0109 07:20:44.657894 2397 tablet_bootstrap.cc:636] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704: Renaming log recovery dir from /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery to /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery-1483975244657893 I0109 07:20:44.657930 2397 tablet_bootstrap.cc:646] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704: Deleting all files from renamed log recovery directory /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery-1483975244657893 I0109 07:20:44.658051 2397 tablet_bootstrap.cc:649] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704: Completed deletion of old log recovery files and directory /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery-1483975244657893 I0109 07:20:44.658063 2397 tablet_bootstrap.cc:381] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704: Bootstrap complete. I0109 07:20:44.663074 2397 ts_tablet_manager.cc:730] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704: Time spent bootstrapping tablet: real 0.058s user 0.016s sys 0.028s I0109 07:20:44.663616 2397 raft_consensus.cc:287] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [term 4 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "710c236995db46699eb9055c19ac506e" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-4.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "1678276a36f849c28459de6b6788965a" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-5.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "591b1a1d383d4b659d7549653f4f136e" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-2.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "c5a3e756029444b2bf66ee0044f5bc35" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-1.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "7897578c54854ba0a5af32eb2aa04704" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-3.gce.cloudera.com" port: 7050 } } I0109 07:20:44.663643 2397 raft_consensus.cc:547] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [term 4 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7897578c54854ba0a5af32eb2aa04704, State: 1, Role: FOLLOWER I0109 07:20:44.663678 2397 consensus_queue.cc:175] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 61, Last appended: 4.61, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER I0109 07:20:44.663846 2397 ts_tablet_manager.cc:757] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s I0109 07:20:44.671183 2383 rpc_server.cc:164] RPC server started. Bound to: 0.0.0.0:7050 I0109 07:20:44.671270 2383 webserver.cc:126] Starting webserver on 0.0.0.0:8050 I0109 07:20:44.671283 2383 webserver.cc:137] Document root disabled I0109 07:20:44.671438 2383 webserver.cc:221] Webserver started. Bound to: http://0.0.0.0:8050/ I0109 07:20:44.671871 2383 tablet_server_main.cc:69] Tablet server successfully started. I0109 07:20:44.706390 2463 raft_consensus.cc:2158] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [term 4 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 0.205s I0109 07:20:44.706435 2463 raft_consensus.cc:1744] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 591b1a1d383d4b659d7549653f4f136e in term 4. I0109 07:20:44.723410 2506 heartbeater.cc:292] Connected to a master server at aserbin-kudu-jepsen-master.gce.cloudera.com:7051 I0109 07:20:44.723459 2506 heartbeater.cc:359] Registering TS with master... I0109 07:20:44.723531 2506 heartbeater.cc:373] Master aserbin-kudu-jepsen-master.gce.cloudera.com:7051 requested a full tablet report, sending... I0109 07:20:44.737507 2463 raft_consensus.cc:1691] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c5a3e756029444b2bf66ee0044f5bc35 for term 5 because replica has last-logged OpId of term: 4 index: 61, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 49. I0109 07:20:44.903971 2463 raft_consensus.cc:2202] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [term 4 FOLLOWER]: Advancing to term 5 I0109 07:20:44.904047 2463 raft_consensus.cc:2158] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [term 5 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 0.165s I0109 07:20:44.904417 2463 raft_consensus.cc:1744] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 591b1a1d383d4b659d7549653f4f136e in term 5. I0109 07:20:44.905154 2463 raft_consensus.cc:1674] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 710c236995db46699eb9055c19ac506e in current term 5: Already voted for candidate 591b1a1d383d4b659d7549653f4f136e in this term. I0109 07:20:45.105165 2463 raft_consensus.cc:915] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [term 5 FOLLOWER]: Refusing update from remote peer 710c236995db46699eb9055c19ac506e: Log matching property violated. Preceding OpId in replica: term: 4 index: 61. Preceding OpId from leader: term: 5 index: 62. (index mismatch) I0109 07:20:45.105904 2506 heartbeater.cc:365] Master aserbin-kudu-jepsen-master.gce.cloudera.com:7051 was elected leader, sending a full tablet report... I0109 07:21:06.428102 2537 tablet_server_main.cc:56] Tablet server non-default flags: --raft_heartbeat_interval_ms=50 --fs_data_dirs=/var/lib/kudu/tserver --fs_wal_dir=/var/lib/kudu/tserver --tserver_master_addrs=aserbin-kudu-jepsen-master.gce.cloudera.com:7051 --heap_profile_path=/tmp/kudu-tserver.2537 --unlock_experimental_flags=true --unlock_unsafe_flags=true --flagfile=/etc/kudu/conf/tserver.gflagfile --log_dir=/var/log/kudu --logtostderr=true Tablet server version: kudu 1.3.0-SNAPSHOT revision 3a23b058af0f4de04f8827255de405fff7bf9ee1 build type RELEASE built by jenkins at 09 Jan 2017 06:07:18 PST on aserbin-kudu-jepsen-control-node-debian8.gce.cloudera.com build id 2017-01-09_06-07-06 I0109 07:21:06.428272 2537 mem_tracker.cc:140] MemTracker: hard memory limit is 0.464404 GB I0109 07:21:06.428282 2537 mem_tracker.cc:142] MemTracker: soft memory limit is 0.278643 GB I0109 07:21:06.429569 2537 tablet_server_main.cc:63] Initializing tablet server... I0109 07:21:06.479689 2537 hybrid_clock.cc:177] HybridClock initialized. Resolution in nanos?: 1 Wait times tolerance adjustment: 1.0005 Current error: 559196 I0109 07:21:06.480428 2537 env_posix.cc:1286] Not raising process file limit of 65536; it is already as high as it can go I0109 07:21:06.480454 2537 file_cache.cc:401] Constructed file cache lbm with capacity 32768 I0109 07:21:06.481484 2537 fs_manager.cc:251] Opened local filesystem: /var/lib/kudu/tserver uuid: "7897578c54854ba0a5af32eb2aa04704" format_stamp: "Formatted at 2017-01-09 15:20:11 on aserbin-kudu-jepsen-ts-3" I0109 07:21:06.484567 2537 ts_tablet_manager.cc:701] Loading metadata for tablet 9e99b710634a44f4a948132a5a180600 I0109 07:21:06.484776 2537 ts_tablet_manager.cc:863] Registered tablet 9e99b710634a44f4a948132a5a180600 I0109 07:21:06.484974 2551 ts_tablet_manager.cc:725] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704: Bootstrapping tablet I0109 07:21:06.485175 2551 tablet_bootstrap.cc:381] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704: Bootstrap starting. I0109 07:21:06.485435 2551 tablet_bootstrap.cc:540] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704: Time spent opening tablet: real 0.000s user 0.000s sys 0.000s I0109 07:21:06.485514 2551 tablet_bootstrap.cc:596] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704: Will attempt to recover log segment /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600/wal-000000001 to /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery/wal-000000001 I0109 07:21:06.485540 2551 tablet_bootstrap.cc:604] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704: Moving log directory /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600 to recovery directory /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery in preparation for log replay I0109 07:21:06.485730 2551 log_util.cc:315] Log segment /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery/wal-000000001 has no footer. This segment was likely being written when the server previously shut down. I0109 07:21:06.485740 2551 log_reader.cc:160] Log segment /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery/wal-000000001 was likely left in-progress after a previous crash. Will try to rebuild footer by scanning data. I0109 07:21:06.487279 2537 tablet_server_main.cc:66] Starting tablet server... I0109 07:21:06.488926 2551 log_util.cc:569] Scanning /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery/wal-000000001 for valid entry headers following offset 36126... I0109 07:21:06.554841 2551 log_util.cc:606] Found no log entry headers I0109 07:21:06.554903 2551 log_util.cc:218] Ignoring log segment corruption in /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery/wal-000000001 because there are no log entries following the corrupted one. The server probably crashed in the middle of writing an entry to the write-ahead log or downloaded an active log via tablet copy. Error detail: Corruption: CRC mismatch in log entry header: Log file corruption detected. Failed trying to read batch #0 at offset 36114 for log segment /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery/wal-000000001: Prior entries: [off=35846 REPLICATE (5.157)] [off=35881 COMMIT (5.157)] [off=36079 REPLICATE (5.158)] [off=36114 COMMIT (5.158)] I0109 07:21:06.554927 2551 log_util.cc:368] Successfully rebuilt footer for segment: /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery/wal-000000001 (valid entries through byte offset 36114) I0109 07:21:06.554989 2551 tablet.cc:983] T 9e99b710634a44f4a948132a5a180600 Rewinding schema during bootstrap to Schema [ 0:key[string NOT NULL], 1:value[int32 NOT NULL] ] I0109 07:21:06.555204 2551 log.cc:351] Log is configured to *not* fsync() on all Append() calls I0109 07:21:06.565366 2551 tablet_bootstrap.cc:381] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704: Bootstrap replayed 1/1 log segments. Stats: ops{read=158 overwritten=0 applied=158 ignored=0} inserts{seen=155 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I0109 07:21:06.565400 2551 tablet_bootstrap.cc:1017] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704: ReplayState: Previous OpId: 5.158, Committed OpId: 5.158, Pending Replicates: 0, Pending Commits: 0 I0109 07:21:06.565831 2551 tablet_bootstrap.cc:633] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704: Preparing to delete log recovery files and directory /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery I0109 07:21:06.565856 2551 tablet_bootstrap.cc:636] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704: Renaming log recovery dir from /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery to /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery-1483975266565855 I0109 07:21:06.565876 2551 tablet_bootstrap.cc:646] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704: Deleting all files from renamed log recovery directory /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery-1483975266565855 I0109 07:21:06.565995 2551 tablet_bootstrap.cc:649] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704: Completed deletion of old log recovery files and directory /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery-1483975266565855 I0109 07:21:06.566007 2551 tablet_bootstrap.cc:381] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704: Bootstrap complete. I0109 07:21:06.571101 2551 ts_tablet_manager.cc:730] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704: Time spent bootstrapping tablet: real 0.086s user 0.012s sys 0.036s I0109 07:21:06.571708 2551 raft_consensus.cc:287] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [term 5 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "710c236995db46699eb9055c19ac506e" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-4.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "1678276a36f849c28459de6b6788965a" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-5.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "591b1a1d383d4b659d7549653f4f136e" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-2.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "c5a3e756029444b2bf66ee0044f5bc35" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-1.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "7897578c54854ba0a5af32eb2aa04704" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-3.gce.cloudera.com" port: 7050 } } I0109 07:21:06.571732 2551 raft_consensus.cc:547] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [term 5 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7897578c54854ba0a5af32eb2aa04704, State: 1, Role: FOLLOWER I0109 07:21:06.571744 2551 consensus_queue.cc:175] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 158, Last appended: 5.158, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER I0109 07:21:06.571904 2551 ts_tablet_manager.cc:757] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s I0109 07:21:06.579746 2537 rpc_server.cc:164] RPC server started. Bound to: 0.0.0.0:7050 I0109 07:21:06.579849 2537 webserver.cc:126] Starting webserver on 0.0.0.0:8050 I0109 07:21:06.579865 2537 webserver.cc:137] Document root disabled I0109 07:21:06.580041 2537 webserver.cc:221] Webserver started. Bound to: http://0.0.0.0:8050/ I0109 07:21:06.580431 2537 tablet_server_main.cc:69] Tablet server successfully started. I0109 07:21:06.632771 2667 heartbeater.cc:292] Connected to a master server at aserbin-kudu-jepsen-master.gce.cloudera.com:7051 I0109 07:21:06.632818 2667 heartbeater.cc:359] Registering TS with master... I0109 07:21:06.632894 2667 heartbeater.cc:373] Master aserbin-kudu-jepsen-master.gce.cloudera.com:7051 requested a full tablet report, sending... I0109 07:21:06.805024 2589 raft_consensus.cc:410] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0109 07:21:06.805145 2589 raft_consensus.cc:2158] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [term 5 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 0.156s I0109 07:21:06.805232 2589 raft_consensus.cc:434] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "710c236995db46699eb9055c19ac506e" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-4.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "1678276a36f849c28459de6b6788965a" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-5.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "591b1a1d383d4b659d7549653f4f136e" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-2.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "c5a3e756029444b2bf66ee0044f5bc35" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-1.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "7897578c54854ba0a5af32eb2aa04704" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-3.gce.cloudera.com" port: 7050 } } I0109 07:21:07.004173 2589 leader_election.cc:215] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [CANDIDATE]: Term 6 pre-election: Requesting pre-vote from peer 710c236995db46699eb9055c19ac506e I0109 07:21:07.004405 2589 leader_election.cc:215] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [CANDIDATE]: Term 6 pre-election: Requesting pre-vote from peer 1678276a36f849c28459de6b6788965a I0109 07:21:07.004535 2589 leader_election.cc:215] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [CANDIDATE]: Term 6 pre-election: Requesting pre-vote from peer 591b1a1d383d4b659d7549653f4f136e I0109 07:21:07.004618 2589 leader_election.cc:215] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [CANDIDATE]: Term 6 pre-election: Requesting pre-vote from peer c5a3e756029444b2bf66ee0044f5bc35 W0109 07:21:07.005610 2549 leader_election.cc:271] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 591b1a1d383d4b659d7549653f4f136e: Network error: Client connection negotiation failed: client connection to 172.31.113.2:7050: connect: Connection refused (error 111) W0109 07:21:07.005784 2547 leader_election.cc:271] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 1678276a36f849c28459de6b6788965a: Network error: Client connection negotiation failed: client connection to 172.31.114.39:7050: connect: Connection refused (error 111) W0109 07:21:07.005924 2546 leader_election.cc:271] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer c5a3e756029444b2bf66ee0044f5bc35: Network error: Client connection negotiation failed: client connection to 172.31.112.255:7050: connect: Connection refused (error 111) I0109 07:21:07.005945 2546 leader_election.cc:242] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. I0109 07:21:07.006270 2673 raft_consensus.cc:2158] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [term 5 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 0.191s I0109 07:21:07.006288 2673 raft_consensus.cc:1905] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority W0109 07:21:07.160679 2548 leader_election.cc:271] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 710c236995db46699eb9055c19ac506e: Timed out: RequestConsensusVote RPC to 172.31.114.38:7050 timed out after 0.156s (SENT) I0109 07:21:07.175740 2624 raft_consensus.cc:2158] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [term 5 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 0.329s I0109 07:21:07.175787 2624 raft_consensus.cc:1744] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 710c236995db46699eb9055c19ac506e in term 5. I0109 07:21:07.374325 2624 raft_consensus.cc:2202] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [term 5 FOLLOWER]: Advancing to term 6 I0109 07:21:07.374421 2624 raft_consensus.cc:2158] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [term 6 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 0.303s I0109 07:21:07.374807 2624 raft_consensus.cc:1744] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 710c236995db46699eb9055c19ac506e in term 6. I0109 07:21:07.575884 2624 raft_consensus.cc:915] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [term 6 FOLLOWER]: Refusing update from remote peer 710c236995db46699eb9055c19ac506e: Log matching property violated. Preceding OpId in replica: term: 5 index: 158. Preceding OpId from leader: term: 6 index: 162. (index mismatch) I0109 07:21:07.576942 2667 heartbeater.cc:365] Master aserbin-kudu-jepsen-master.gce.cloudera.com:7051 was elected leader, sending a full tablet report... I0109 07:21:19.964504 2713 tablet_server_main.cc:56] Tablet server non-default flags: --raft_heartbeat_interval_ms=50 --fs_data_dirs=/var/lib/kudu/tserver --fs_wal_dir=/var/lib/kudu/tserver --tserver_master_addrs=aserbin-kudu-jepsen-master.gce.cloudera.com:7051 --heap_profile_path=/tmp/kudu-tserver.2713 --unlock_experimental_flags=true --unlock_unsafe_flags=true --flagfile=/etc/kudu/conf/tserver.gflagfile --log_dir=/var/log/kudu --logtostderr=true Tablet server version: kudu 1.3.0-SNAPSHOT revision 3a23b058af0f4de04f8827255de405fff7bf9ee1 build type RELEASE built by jenkins at 09 Jan 2017 06:07:18 PST on aserbin-kudu-jepsen-control-node-debian8.gce.cloudera.com build id 2017-01-09_06-07-06 I0109 07:21:19.964653 2713 mem_tracker.cc:140] MemTracker: hard memory limit is 0.464404 GB I0109 07:21:19.964663 2713 mem_tracker.cc:142] MemTracker: soft memory limit is 0.278643 GB I0109 07:21:19.965894 2713 tablet_server_main.cc:63] Initializing tablet server... I0109 07:21:20.015496 2713 hybrid_clock.cc:177] HybridClock initialized. Resolution in nanos?: 1 Wait times tolerance adjustment: 1.0005 Current error: 566196 I0109 07:21:20.016317 2713 env_posix.cc:1286] Not raising process file limit of 65536; it is already as high as it can go I0109 07:21:20.016346 2713 file_cache.cc:401] Constructed file cache lbm with capacity 32768 I0109 07:21:20.017452 2713 fs_manager.cc:251] Opened local filesystem: /var/lib/kudu/tserver uuid: "7897578c54854ba0a5af32eb2aa04704" format_stamp: "Formatted at 2017-01-09 15:20:11 on aserbin-kudu-jepsen-ts-3" I0109 07:21:20.021070 2713 ts_tablet_manager.cc:701] Loading metadata for tablet 9e99b710634a44f4a948132a5a180600 I0109 07:21:20.021360 2713 ts_tablet_manager.cc:863] Registered tablet 9e99b710634a44f4a948132a5a180600 I0109 07:21:20.021528 2727 ts_tablet_manager.cc:725] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704: Bootstrapping tablet I0109 07:21:20.021788 2727 tablet_bootstrap.cc:381] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704: Bootstrap starting. I0109 07:21:20.022086 2727 tablet_bootstrap.cc:540] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704: Time spent opening tablet: real 0.000s user 0.000s sys 0.000s I0109 07:21:20.022166 2727 tablet_bootstrap.cc:596] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704: Will attempt to recover log segment /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600/wal-000000001 to /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery/wal-000000001 I0109 07:21:20.022178 2727 tablet_bootstrap.cc:604] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704: Moving log directory /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600 to recovery directory /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery in preparation for log replay I0109 07:21:20.022402 2727 log_util.cc:315] Log segment /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery/wal-000000001 has no footer. This segment was likely being written when the server previously shut down. I0109 07:21:20.022414 2727 log_reader.cc:160] Log segment /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery/wal-000000001 was likely left in-progress after a previous crash. Will try to rebuild footer by scanning data. I0109 07:21:20.023844 2713 tablet_server_main.cc:66] Starting tablet server... I0109 07:21:20.026522 2727 log_util.cc:569] Scanning /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery/wal-000000001 for valid entry headers following offset 47107... I0109 07:21:20.091594 2727 log_util.cc:606] Found no log entry headers I0109 07:21:20.091657 2727 log_util.cc:218] Ignoring log segment corruption in /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery/wal-000000001 because there are no log entries following the corrupted one. The server probably crashed in the middle of writing an entry to the write-ahead log or downloaded an active log via tablet copy. Error detail: Corruption: CRC mismatch in log entry header: Log file corruption detected. Failed trying to read batch #0 at offset 47095 for log segment /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery/wal-000000001: Prior entries: [off=46827 REPLICATE (6.205)] [off=46862 COMMIT (6.205)] [off=47060 REPLICATE (6.206)] [off=47095 COMMIT (6.206)] I0109 07:21:20.091670 2727 log_util.cc:368] Successfully rebuilt footer for segment: /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery/wal-000000001 (valid entries through byte offset 47095) I0109 07:21:20.091734 2727 tablet.cc:983] T 9e99b710634a44f4a948132a5a180600 Rewinding schema during bootstrap to Schema [ 0:key[string NOT NULL], 1:value[int32 NOT NULL] ] I0109 07:21:20.092100 2727 log.cc:351] Log is configured to *not* fsync() on all Append() calls I0109 07:21:20.105809 2727 tablet_bootstrap.cc:381] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704: Bootstrap replayed 1/1 log segments. Stats: ops{read=206 overwritten=0 applied=206 ignored=0} inserts{seen=202 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I0109 07:21:20.105842 2727 tablet_bootstrap.cc:1017] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704: ReplayState: Previous OpId: 6.206, Committed OpId: 6.206, Pending Replicates: 0, Pending Commits: 0 I0109 07:21:20.106325 2727 tablet_bootstrap.cc:633] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704: Preparing to delete log recovery files and directory /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery I0109 07:21:20.106338 2727 tablet_bootstrap.cc:636] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704: Renaming log recovery dir from /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery to /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery-1483975280106337 I0109 07:21:20.106381 2727 tablet_bootstrap.cc:646] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704: Deleting all files from renamed log recovery directory /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery-1483975280106337 I0109 07:21:20.106524 2727 tablet_bootstrap.cc:649] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704: Completed deletion of old log recovery files and directory /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery-1483975280106337 I0109 07:21:20.106536 2727 tablet_bootstrap.cc:381] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704: Bootstrap complete. I0109 07:21:20.111675 2727 ts_tablet_manager.cc:730] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704: Time spent bootstrapping tablet: real 0.090s user 0.016s sys 0.036s I0109 07:21:20.112149 2727 raft_consensus.cc:287] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [term 6 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "710c236995db46699eb9055c19ac506e" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-4.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "1678276a36f849c28459de6b6788965a" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-5.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "591b1a1d383d4b659d7549653f4f136e" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-2.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "c5a3e756029444b2bf66ee0044f5bc35" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-1.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "7897578c54854ba0a5af32eb2aa04704" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-3.gce.cloudera.com" port: 7050 } } I0109 07:21:20.112190 2727 raft_consensus.cc:547] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [term 6 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7897578c54854ba0a5af32eb2aa04704, State: 1, Role: FOLLOWER I0109 07:21:20.112203 2727 consensus_queue.cc:175] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 206, Last appended: 6.206, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER I0109 07:21:20.112373 2727 ts_tablet_manager.cc:757] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s I0109 07:21:20.119556 2713 rpc_server.cc:164] RPC server started. Bound to: 0.0.0.0:7050 I0109 07:21:20.119639 2713 webserver.cc:126] Starting webserver on 0.0.0.0:8050 I0109 07:21:20.119647 2713 webserver.cc:137] Document root disabled I0109 07:21:20.119802 2713 webserver.cc:221] Webserver started. Bound to: http://0.0.0.0:8050/ I0109 07:21:20.120286 2713 tablet_server_main.cc:69] Tablet server successfully started. I0109 07:21:20.171649 2843 heartbeater.cc:292] Connected to a master server at aserbin-kudu-jepsen-master.gce.cloudera.com:7051 I0109 07:21:20.171694 2843 heartbeater.cc:359] Registering TS with master... I0109 07:21:20.171764 2843 heartbeater.cc:373] Master aserbin-kudu-jepsen-master.gce.cloudera.com:7051 requested a full tablet report, sending... I0109 07:21:20.293447 2769 raft_consensus.cc:410] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0109 07:21:20.293531 2769 raft_consensus.cc:2158] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [term 6 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 0.200s I0109 07:21:20.293602 2769 raft_consensus.cc:434] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "710c236995db46699eb9055c19ac506e" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-4.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "1678276a36f849c28459de6b6788965a" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-5.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "591b1a1d383d4b659d7549653f4f136e" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-2.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "c5a3e756029444b2bf66ee0044f5bc35" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-1.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "7897578c54854ba0a5af32eb2aa04704" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-3.gce.cloudera.com" port: 7050 } } I0109 07:21:20.489832 2769 leader_election.cc:215] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [CANDIDATE]: Term 7 pre-election: Requesting pre-vote from peer 710c236995db46699eb9055c19ac506e I0109 07:21:20.490057 2769 leader_election.cc:215] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [CANDIDATE]: Term 7 pre-election: Requesting pre-vote from peer 1678276a36f849c28459de6b6788965a I0109 07:21:20.490195 2769 leader_election.cc:215] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [CANDIDATE]: Term 7 pre-election: Requesting pre-vote from peer 591b1a1d383d4b659d7549653f4f136e I0109 07:21:20.490299 2769 leader_election.cc:215] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [CANDIDATE]: Term 7 pre-election: Requesting pre-vote from peer c5a3e756029444b2bf66ee0044f5bc35 W0109 07:21:20.491282 2724 leader_election.cc:271] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 710c236995db46699eb9055c19ac506e: Network error: Client connection negotiation failed: client connection to 172.31.114.38:7050: connect: Connection refused (error 111) W0109 07:21:20.491371 2725 leader_election.cc:271] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 591b1a1d383d4b659d7549653f4f136e: Network error: Client connection negotiation failed: client connection to 172.31.113.2:7050: connect: Connection refused (error 111) W0109 07:21:20.491480 2722 leader_election.cc:271] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer c5a3e756029444b2bf66ee0044f5bc35: Network error: Client connection negotiation failed: client connection to 172.31.112.255:7050: connect: Connection refused (error 111) I0109 07:21:20.491498 2722 leader_election.cc:242] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. I0109 07:21:20.491677 2849 raft_consensus.cc:2158] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [term 6 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 0.157s I0109 07:21:20.491693 2849 raft_consensus.cc:1905] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority I0109 07:21:20.492810 2723 leader_election.cc:361] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [CANDIDATE]: Term 7 pre-election: Vote denied by peer 1678276a36f849c28459de6b6788965a. Message: Invalid argument: T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7897578c54854ba0a5af32eb2aa04704 for term 7 because replica has last-logged OpId of term: 6 index: 209, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 206. I0109 07:21:20.719446 2800 raft_consensus.cc:2158] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [term 6 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 0.258s I0109 07:21:20.719506 2800 raft_consensus.cc:1744] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1678276a36f849c28459de6b6788965a in term 6. I0109 07:21:21.129801 2769 raft_consensus.cc:410] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0109 07:21:21.129873 2769 raft_consensus.cc:2158] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [term 6 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 0.277s I0109 07:21:21.129932 2769 raft_consensus.cc:434] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "710c236995db46699eb9055c19ac506e" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-4.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "1678276a36f849c28459de6b6788965a" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-5.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "591b1a1d383d4b659d7549653f4f136e" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-2.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "c5a3e756029444b2bf66ee0044f5bc35" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-1.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "7897578c54854ba0a5af32eb2aa04704" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-3.gce.cloudera.com" port: 7050 } } I0109 07:21:21.174383 2843 heartbeater.cc:365] Master aserbin-kudu-jepsen-master.gce.cloudera.com:7051 was elected leader, sending a full tablet report... I0109 07:21:21.326530 2800 raft_consensus.cc:2158] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [term 6 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 0.188s I0109 07:21:21.326581 2800 raft_consensus.cc:1744] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1678276a36f849c28459de6b6788965a in term 6. I0109 07:21:21.326692 2769 leader_election.cc:215] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [CANDIDATE]: Term 7 pre-election: Requesting pre-vote from peer 710c236995db46699eb9055c19ac506e I0109 07:21:21.326853 2769 leader_election.cc:215] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [CANDIDATE]: Term 7 pre-election: Requesting pre-vote from peer 1678276a36f849c28459de6b6788965a I0109 07:21:21.326913 2769 leader_election.cc:215] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [CANDIDATE]: Term 7 pre-election: Requesting pre-vote from peer 591b1a1d383d4b659d7549653f4f136e I0109 07:21:21.327018 2769 leader_election.cc:215] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [CANDIDATE]: Term 7 pre-election: Requesting pre-vote from peer c5a3e756029444b2bf66ee0044f5bc35 I0109 07:21:21.327447 2723 leader_election.cc:361] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [CANDIDATE]: Term 7 pre-election: Vote denied by peer 1678276a36f849c28459de6b6788965a. Message: Invalid argument: T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7897578c54854ba0a5af32eb2aa04704 for term 7 because replica has last-logged OpId of term: 6 index: 209, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 206. W0109 07:21:21.327687 2724 leader_election.cc:271] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 710c236995db46699eb9055c19ac506e: Network error: Client connection negotiation failed: client connection to 172.31.114.38:7050: connect: Connection refused (error 111) W0109 07:21:21.327728 2725 leader_election.cc:271] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 591b1a1d383d4b659d7549653f4f136e: Network error: Client connection negotiation failed: client connection to 172.31.113.2:7050: connect: Connection refused (error 111) I0109 07:21:21.327738 2725 leader_election.cc:242] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. W0109 07:21:21.327769 2722 leader_election.cc:271] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer c5a3e756029444b2bf66ee0044f5bc35: Network error: Client connection negotiation failed: client connection to 172.31.112.255:7050: connect: Connection refused (error 111) I0109 07:21:21.327935 2861 raft_consensus.cc:2158] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [term 6 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 0.236s I0109 07:21:21.327955 2861 raft_consensus.cc:1905] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority I0109 07:21:21.730193 2769 raft_consensus.cc:410] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0109 07:21:21.730295 2769 raft_consensus.cc:2158] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [term 6 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 0.222s I0109 07:21:21.730394 2769 raft_consensus.cc:434] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "710c236995db46699eb9055c19ac506e" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-4.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "1678276a36f849c28459de6b6788965a" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-5.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "591b1a1d383d4b659d7549653f4f136e" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-2.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "c5a3e756029444b2bf66ee0044f5bc35" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-1.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "7897578c54854ba0a5af32eb2aa04704" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-3.gce.cloudera.com" port: 7050 } } I0109 07:21:21.927371 2800 raft_consensus.cc:2158] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [term 6 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 0.292s I0109 07:21:21.927419 2800 raft_consensus.cc:1744] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1678276a36f849c28459de6b6788965a in term 6. I0109 07:21:21.927577 2769 leader_election.cc:215] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [CANDIDATE]: Term 7 pre-election: Requesting pre-vote from peer 710c236995db46699eb9055c19ac506e I0109 07:21:21.927728 2769 leader_election.cc:215] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [CANDIDATE]: Term 7 pre-election: Requesting pre-vote from peer 1678276a36f849c28459de6b6788965a I0109 07:21:21.927809 2769 leader_election.cc:215] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [CANDIDATE]: Term 7 pre-election: Requesting pre-vote from peer 591b1a1d383d4b659d7549653f4f136e I0109 07:21:21.927899 2769 leader_election.cc:215] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [CANDIDATE]: Term 7 pre-election: Requesting pre-vote from peer c5a3e756029444b2bf66ee0044f5bc35 I0109 07:21:21.928289 2723 leader_election.cc:361] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [CANDIDATE]: Term 7 pre-election: Vote denied by peer 1678276a36f849c28459de6b6788965a. Message: Invalid argument: T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7897578c54854ba0a5af32eb2aa04704 for term 7 because replica has last-logged OpId of term: 6 index: 209, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 206. W0109 07:21:21.928632 2724 leader_election.cc:271] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 710c236995db46699eb9055c19ac506e: Network error: Client connection negotiation failed: client connection to 172.31.114.38:7050: connect: Connection refused (error 111) W0109 07:21:21.928678 2725 leader_election.cc:271] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 591b1a1d383d4b659d7549653f4f136e: Network error: Client connection negotiation failed: client connection to 172.31.113.2:7050: connect: Connection refused (error 111) I0109 07:21:21.928689 2725 leader_election.cc:242] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. W0109 07:21:21.928735 2722 leader_election.cc:271] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer c5a3e756029444b2bf66ee0044f5bc35: Network error: Client connection negotiation failed: client connection to 172.31.112.255:7050: connect: Connection refused (error 111) I0109 07:21:21.928885 2865 raft_consensus.cc:2158] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [term 6 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 0.413s I0109 07:21:21.928900 2865 raft_consensus.cc:1905] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority I0109 07:21:22.465050 2800 raft_consensus.cc:2158] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [term 6 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 0.557s I0109 07:21:22.465107 2800 raft_consensus.cc:1744] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1678276a36f849c28459de6b6788965a in term 6. I0109 07:21:23.221227 2769 raft_consensus.cc:410] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0109 07:21:23.221313 2769 raft_consensus.cc:2158] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [term 6 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 0.514s I0109 07:21:23.221393 2769 raft_consensus.cc:434] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "710c236995db46699eb9055c19ac506e" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-4.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "1678276a36f849c28459de6b6788965a" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-5.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "591b1a1d383d4b659d7549653f4f136e" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-2.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "c5a3e756029444b2bf66ee0044f5bc35" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-1.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "7897578c54854ba0a5af32eb2aa04704" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-3.gce.cloudera.com" port: 7050 } } I0109 07:21:23.417882 2769 leader_election.cc:215] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [CANDIDATE]: Term 7 pre-election: Requesting pre-vote from peer 710c236995db46699eb9055c19ac506e I0109 07:21:23.418146 2769 leader_election.cc:215] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [CANDIDATE]: Term 7 pre-election: Requesting pre-vote from peer 1678276a36f849c28459de6b6788965a I0109 07:21:23.418242 2769 leader_election.cc:215] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [CANDIDATE]: Term 7 pre-election: Requesting pre-vote from peer 591b1a1d383d4b659d7549653f4f136e I0109 07:21:23.418332 2769 leader_election.cc:215] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [CANDIDATE]: Term 7 pre-election: Requesting pre-vote from peer c5a3e756029444b2bf66ee0044f5bc35 W0109 07:21:23.418943 2724 leader_election.cc:271] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 710c236995db46699eb9055c19ac506e: Network error: Client connection negotiation failed: client connection to 172.31.114.38:7050: connect: Connection refused (error 111) W0109 07:21:23.418987 2725 leader_election.cc:271] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 591b1a1d383d4b659d7549653f4f136e: Network error: Client connection negotiation failed: client connection to 172.31.113.2:7050: connect: Connection refused (error 111) W0109 07:21:23.419030 2722 leader_election.cc:271] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer c5a3e756029444b2bf66ee0044f5bc35: Network error: Client connection negotiation failed: client connection to 172.31.112.255:7050: connect: Connection refused (error 111) I0109 07:21:23.419040 2722 leader_election.cc:242] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. I0109 07:21:23.419188 2869 raft_consensus.cc:2158] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [term 6 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 0.320s I0109 07:21:23.419204 2869 raft_consensus.cc:1905] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority I0109 07:21:23.427390 2723 leader_election.cc:361] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [CANDIDATE]: Term 7 pre-election: Vote denied by peer 1678276a36f849c28459de6b6788965a. Message: Invalid argument: T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7897578c54854ba0a5af32eb2aa04704 for term 7 because replica has last-logged OpId of term: 6 index: 209, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 206. I0109 07:21:23.428050 2800 raft_consensus.cc:2158] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [term 6 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 0.628s I0109 07:21:23.428077 2800 raft_consensus.cc:1744] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1678276a36f849c28459de6b6788965a in term 6. I0109 07:21:24.213644 2769 raft_consensus.cc:410] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0109 07:21:24.213721 2769 raft_consensus.cc:2158] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [term 6 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 0.327s I0109 07:21:24.213784 2769 raft_consensus.cc:434] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "710c236995db46699eb9055c19ac506e" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-4.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "1678276a36f849c28459de6b6788965a" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-5.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "591b1a1d383d4b659d7549653f4f136e" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-2.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "c5a3e756029444b2bf66ee0044f5bc35" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-1.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "7897578c54854ba0a5af32eb2aa04704" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-3.gce.cloudera.com" port: 7050 } } I0109 07:21:24.411000 2769 leader_election.cc:215] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [CANDIDATE]: Term 7 pre-election: Requesting pre-vote from peer 710c236995db46699eb9055c19ac506e I0109 07:21:24.411249 2769 leader_election.cc:215] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [CANDIDATE]: Term 7 pre-election: Requesting pre-vote from peer 1678276a36f849c28459de6b6788965a I0109 07:21:24.411356 2769 leader_election.cc:215] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [CANDIDATE]: Term 7 pre-election: Requesting pre-vote from peer 591b1a1d383d4b659d7549653f4f136e I0109 07:21:24.411481 2769 leader_election.cc:215] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [CANDIDATE]: Term 7 pre-election: Requesting pre-vote from peer c5a3e756029444b2bf66ee0044f5bc35 W0109 07:21:24.412111 2724 leader_election.cc:271] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 710c236995db46699eb9055c19ac506e: Network error: Client connection negotiation failed: client connection to 172.31.114.38:7050: connect: Connection refused (error 111) I0109 07:21:24.412169 2723 leader_election.cc:361] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [CANDIDATE]: Term 7 pre-election: Vote denied by peer 1678276a36f849c28459de6b6788965a. Message: Invalid argument: T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7897578c54854ba0a5af32eb2aa04704 for term 7 because replica has last-logged OpId of term: 6 index: 209, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 206. W0109 07:21:24.412245 2725 leader_election.cc:271] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 591b1a1d383d4b659d7549653f4f136e: Network error: Client connection negotiation failed: client connection to 172.31.113.2:7050: connect: Connection refused (error 111) I0109 07:21:24.412253 2725 leader_election.cc:242] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. W0109 07:21:24.412286 2722 leader_election.cc:271] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer c5a3e756029444b2bf66ee0044f5bc35: Network error: Client connection negotiation failed: client connection to 172.31.112.255:7050: connect: Connection refused (error 111) I0109 07:21:24.412437 2873 raft_consensus.cc:2158] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [term 6 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 0.268s I0109 07:21:24.412452 2873 raft_consensus.cc:1905] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority I0109 07:21:24.725208 2800 raft_consensus.cc:2158] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [term 6 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 0.980s I0109 07:21:24.725278 2800 raft_consensus.cc:1744] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1678276a36f849c28459de6b6788965a in term 6. I0109 07:21:25.810616 2800 raft_consensus.cc:2158] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [term 6 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 0.974s I0109 07:21:25.810662 2800 raft_consensus.cc:1744] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5a3e756029444b2bf66ee0044f5bc35 in term 6. I0109 07:21:25.944671 2800 raft_consensus.cc:2158] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [term 6 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 0.254s I0109 07:21:25.944732 2800 raft_consensus.cc:1744] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 591b1a1d383d4b659d7549653f4f136e in term 6. I0109 07:21:26.009891 2800 raft_consensus.cc:2202] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [term 6 FOLLOWER]: Advancing to term 7 I0109 07:21:26.010027 2800 raft_consensus.cc:2158] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [term 7 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 0.840s I0109 07:21:26.010530 2800 raft_consensus.cc:1744] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5a3e756029444b2bf66ee0044f5bc35 in term 7. I0109 07:21:26.142982 2800 raft_consensus.cc:1674] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 591b1a1d383d4b659d7549653f4f136e in current term 7: Already voted for candidate c5a3e756029444b2bf66ee0044f5bc35 in this term. I0109 07:21:26.212208 2800 raft_consensus.cc:915] T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [term 7 FOLLOWER]: Refusing update from remote peer c5a3e756029444b2bf66ee0044f5bc35: Log matching property violated. Preceding OpId in replica: term: 6 index: 206. Preceding OpId from leader: term: 7 index: 210. (index mismatch)