I0109 07:20:10.237417 1270 tablet_server_main.cc:56] Tablet server non-default flags: --raft_heartbeat_interval_ms=50 --fs_data_dirs=/var/lib/kudu/tserver --fs_wal_dir=/var/lib/kudu/tserver --tserver_master_addrs=aserbin-kudu-jepsen-master.gce.cloudera.com:7051 --heap_profile_path=/tmp/kudu-tserver.1270 --unlock_experimental_flags=true --unlock_unsafe_flags=true --flagfile=/etc/kudu/conf/tserver.gflagfile --log_dir=/var/log/kudu --logtostderr=true Tablet server version: kudu 1.3.0-SNAPSHOT revision 3a23b058af0f4de04f8827255de405fff7bf9ee1 build type RELEASE built by jenkins at 09 Jan 2017 06:07:18 PST on aserbin-kudu-jepsen-control-node-debian8.gce.cloudera.com build id 2017-01-09_06-07-06 I0109 07:20:10.240229 1270 mem_tracker.cc:140] MemTracker: hard memory limit is 0.464404 GB I0109 07:20:10.240281 1270 mem_tracker.cc:142] MemTracker: soft memory limit is 0.278643 GB I0109 07:20:10.275355 1270 tablet_server_main.cc:63] Initializing tablet server... I0109 07:20:10.325331 1270 hybrid_clock.cc:177] HybridClock initialized. Resolution in nanos?: 1 Wait times tolerance adjustment: 1.0005 Current error: 452973 I0109 07:20:10.327706 1270 env_posix.cc:1286] Not raising process file limit of 65536; it is already as high as it can go I0109 07:20:10.327771 1270 file_cache.cc:401] Constructed file cache lbm with capacity 32768 I0109 07:20:10.329560 1270 server_base.cc:179] Could not load existing FS layout: Not found: /var/lib/kudu/tserver/instance: No such file or directory (error 2) I0109 07:20:10.329588 1270 server_base.cc:180] Creating new FS layout I0109 07:20:10.337239 1270 fs_manager.cc:362] Generated new instance metadata in path /var/lib/kudu/tserver/instance: uuid: "1678276a36f849c28459de6b6788965a" format_stamp: "Formatted at 2017-01-09 15:20:10 on aserbin-kudu-jepsen-ts-5" I0109 07:20:10.345371 1270 fs_manager.cc:251] Opened local filesystem: /var/lib/kudu/tserver uuid: "1678276a36f849c28459de6b6788965a" format_stamp: "Formatted at 2017-01-09 15:20:10 on aserbin-kudu-jepsen-ts-5" I0109 07:20:10.350301 1270 tablet_server_main.cc:66] Starting tablet server... I0109 07:20:10.366536 1270 rpc_server.cc:164] RPC server started. Bound to: 0.0.0.0:7050 I0109 07:20:10.366681 1270 webserver.cc:126] Starting webserver on 0.0.0.0:8050 I0109 07:20:10.366694 1270 webserver.cc:137] Document root disabled I0109 07:20:10.366901 1270 webserver.cc:221] Webserver started. Bound to: http://0.0.0.0:8050/ I0109 07:20:10.367426 1270 tablet_server_main.cc:69] Tablet server successfully started. W0109 07:20:10.420745 1420 heartbeater.cc:453] Failed to heartbeat to aserbin-kudu-jepsen-master.gce.cloudera.com:7051: Network error: Failed to ping master at aserbin-kudu-jepsen-master.gce.cloudera.com:7051: Client connection negotiation failed: client connection to 172.31.113.125:7051: connect: Connection refused (error 111) W0109 07:20:10.470530 1420 heartbeater.cc:453] Failed to heartbeat to aserbin-kudu-jepsen-master.gce.cloudera.com:7051: Network error: Failed to ping master at aserbin-kudu-jepsen-master.gce.cloudera.com:7051: Client connection negotiation failed: client connection to 172.31.113.125:7051: connect: Connection refused (error 111) W0109 07:20:10.519930 1420 heartbeater.cc:453] Failed to heartbeat to aserbin-kudu-jepsen-master.gce.cloudera.com:7051: Network error: Failed to ping master at aserbin-kudu-jepsen-master.gce.cloudera.com:7051: Client connection negotiation failed: client connection to 172.31.113.125:7051: connect: Connection refused (error 111) W0109 07:20:10.520005 1420 heartbeater.cc:323] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts. I0109 07:20:11.571100 1420 heartbeater.cc:292] Connected to a master server at aserbin-kudu-jepsen-master.gce.cloudera.com:7051 I0109 07:20:11.571164 1420 heartbeater.cc:359] Registering TS with master... I0109 07:20:11.571247 1420 heartbeater.cc:373] Master aserbin-kudu-jepsen-master.gce.cloudera.com:7051 requested a full tablet report, sending... I0109 07:20:12.571853 1420 heartbeater.cc:365] Master aserbin-kudu-jepsen-master.gce.cloudera.com:7051 was elected leader, sending a full tablet report... I0109 07:20:16.878420 1357 tablet_service.cc:630] Processing CreateTablet for tablet 9e99b710634a44f4a948132a5a180600 (table=rw-register-1483975178870 [id=0e0bbef280744ed1a8036e11eb0aaea0]), partition=RANGE (key) PARTITION UNBOUNDED I0109 07:20:16.885817 1357 ts_tablet_manager.cc:863] Registered tablet 9e99b710634a44f4a948132a5a180600 I0109 07:20:16.886198 1439 ts_tablet_manager.cc:725] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a: Bootstrapping tablet I0109 07:20:16.886404 1439 tablet_bootstrap.cc:381] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a: Bootstrap starting. I0109 07:20:16.887271 1439 tablet_bootstrap.cc:540] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a: Time spent opening tablet: real 0.000s user 0.000s sys 0.000s I0109 07:20:16.887369 1439 tablet_bootstrap.cc:483] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a: No blocks or log segments found. Creating new log. I0109 07:20:16.887492 1439 log.cc:351] Log is configured to *not* fsync() on all Append() calls I0109 07:20:16.887902 1439 tablet_bootstrap.cc:381] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a: No bootstrap required, opened a new log I0109 07:20:16.887925 1439 ts_tablet_manager.cc:730] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.000s I0109 07:20:16.889403 1439 raft_consensus.cc:287] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "710c236995db46699eb9055c19ac506e" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-4.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "1678276a36f849c28459de6b6788965a" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-5.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "591b1a1d383d4b659d7549653f4f136e" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-2.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "c5a3e756029444b2bf66ee0044f5bc35" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-1.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "7897578c54854ba0a5af32eb2aa04704" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-3.gce.cloudera.com" port: 7050 } } I0109 07:20:16.889468 1439 raft_consensus.cc:318] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0109 07:20:16.889488 1439 raft_consensus.cc:547] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1678276a36f849c28459de6b6788965a, State: 1, Role: FOLLOWER I0109 07:20:16.889516 1439 consensus_queue.cc:175] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER I0109 07:20:16.889708 1439 ts_tablet_manager.cc:757] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s I0109 07:20:16.947499 1442 raft_consensus.cc:410] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0109 07:20:16.947625 1442 raft_consensus.cc:2158] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [term 0 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 0.172s I0109 07:20:16.947713 1442 raft_consensus.cc:434] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "710c236995db46699eb9055c19ac506e" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-4.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "1678276a36f849c28459de6b6788965a" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-5.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "591b1a1d383d4b659d7549653f4f136e" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-2.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "c5a3e756029444b2bf66ee0044f5bc35" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-1.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "7897578c54854ba0a5af32eb2aa04704" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-3.gce.cloudera.com" port: 7050 } } I0109 07:20:17.146363 1377 raft_consensus.cc:2158] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [term 0 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 0.222s I0109 07:20:17.146421 1377 raft_consensus.cc:1744] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 710c236995db46699eb9055c19ac506e in term 0. I0109 07:20:17.146538 1442 leader_election.cc:215] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 710c236995db46699eb9055c19ac506e I0109 07:20:17.146823 1442 leader_election.cc:215] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 591b1a1d383d4b659d7549653f4f136e I0109 07:20:17.147043 1442 leader_election.cc:215] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer c5a3e756029444b2bf66ee0044f5bc35 I0109 07:20:17.147152 1442 leader_election.cc:215] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 7897578c54854ba0a5af32eb2aa04704 I0109 07:20:17.148349 1313 leader_election.cc:347] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [CANDIDATE]: Term 1 pre-election: Vote granted by peer 710c236995db46699eb9055c19ac506e W0109 07:20:17.149565 1314 leader_election.cc:277] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7897578c54854ba0a5af32eb2aa04704: Not found: Tablet not found I0109 07:20:17.255635 1314 leader_election.cc:361] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [CANDIDATE]: Term 1 pre-election: Vote denied by peer 591b1a1d383d4b659d7549653f4f136e. Message: Service unavailable: T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [term 0 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1678276a36f849c28459de6b6788965a for term 1 because replica is already servicing an update from a current leader or another vote. I0109 07:20:17.257464 1377 raft_consensus.cc:2158] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [term 0 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 0.205s I0109 07:20:17.257483 1377 raft_consensus.cc:1744] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 591b1a1d383d4b659d7549653f4f136e in term 0. W0109 07:20:17.319833 1311 leader_election.cc:271] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer c5a3e756029444b2bf66ee0044f5bc35: Timed out: RequestConsensusVote RPC to 172.31.112.255:7050 timed out after 0.172s (SENT) I0109 07:20:17.319902 1311 leader_election.cc:242] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. I0109 07:20:17.320149 1447 raft_consensus.cc:2158] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [term 0 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 0.167s I0109 07:20:17.320165 1447 raft_consensus.cc:1905] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority I0109 07:20:17.334667 1377 raft_consensus.cc:2158] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [term 0 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 0.311s I0109 07:20:17.334717 1377 raft_consensus.cc:1744] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5a3e756029444b2bf66ee0044f5bc35 in term 0. I0109 07:20:17.422475 1377 raft_consensus.cc:2158] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [term 0 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 0.259s I0109 07:20:17.422525 1377 raft_consensus.cc:1744] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7897578c54854ba0a5af32eb2aa04704 in term 0. I0109 07:20:17.456612 1377 raft_consensus.cc:2202] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [term 0 FOLLOWER]: Advancing to term 1 I0109 07:20:17.456713 1377 raft_consensus.cc:2158] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 0.263s I0109 07:20:17.457139 1377 raft_consensus.cc:1744] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 710c236995db46699eb9055c19ac506e in term 1. I0109 07:20:17.457818 1377 raft_consensus.cc:1674] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 591b1a1d383d4b659d7549653f4f136e in current term 1: Already voted for candidate 710c236995db46699eb9055c19ac506e in this term. I0109 07:20:17.622388 1377 raft_consensus.cc:1674] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 7897578c54854ba0a5af32eb2aa04704 in current term 1: Already voted for candidate 710c236995db46699eb9055c19ac506e in this term. I0109 07:20:17.658516 1377 raft_consensus.cc:915] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [term 1 FOLLOWER]: Refusing update from remote peer 710c236995db46699eb9055c19ac506e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch) I0109 07:20:17.804591 1377 raft_consensus.cc:1705] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 591b1a1d383d4b659d7549653f4f136e for term 2 because replica is either leader or believes a valid leader to be alive. I0109 07:20:24.392710 1442 raft_consensus.cc:410] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 710c236995db46699eb9055c19ac506e) I0109 07:20:24.392807 1442 raft_consensus.cc:2158] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 0.199s I0109 07:20:24.392901 1442 raft_consensus.cc:434] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "710c236995db46699eb9055c19ac506e" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-4.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "1678276a36f849c28459de6b6788965a" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-5.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "591b1a1d383d4b659d7549653f4f136e" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-2.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "c5a3e756029444b2bf66ee0044f5bc35" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-1.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "7897578c54854ba0a5af32eb2aa04704" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-3.gce.cloudera.com" port: 7050 } } I0109 07:20:24.590145 1442 leader_election.cc:215] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 710c236995db46699eb9055c19ac506e I0109 07:20:24.590229 1442 leader_election.cc:215] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 591b1a1d383d4b659d7549653f4f136e I0109 07:20:24.590243 1442 leader_election.cc:215] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer c5a3e756029444b2bf66ee0044f5bc35 I0109 07:20:24.590258 1442 leader_election.cc:215] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 7897578c54854ba0a5af32eb2aa04704 W0109 07:20:24.592454 1314 leader_election.cc:271] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 591b1a1d383d4b659d7549653f4f136e: Network error: Client connection negotiation failed: client connection to 172.31.113.2:7050: connect: Connection refused (error 111) W0109 07:20:24.592674 1313 leader_election.cc:271] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 710c236995db46699eb9055c19ac506e: Network error: Client connection negotiation failed: client connection to 172.31.114.38:7050: connect: Connection refused (error 111) W0109 07:20:24.592758 1311 leader_election.cc:271] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c5a3e756029444b2bf66ee0044f5bc35: Network error: Client connection negotiation failed: client connection to 172.31.112.255:7050: connect: Connection refused (error 111) I0109 07:20:24.592772 1311 leader_election.cc:242] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. W0109 07:20:24.592849 1314 leader_election.cc:271] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 7897578c54854ba0a5af32eb2aa04704: Network error: Client connection negotiation failed: client connection to 172.31.114.37:7050: connect: Connection refused (error 111) I0109 07:20:24.593035 1457 raft_consensus.cc:2158] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 0.197s I0109 07:20:24.593055 1457 raft_consensus.cc:1905] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority I0109 07:20:31.075708 1467 tablet_server_main.cc:56] Tablet server non-default flags: --raft_heartbeat_interval_ms=50 --fs_data_dirs=/var/lib/kudu/tserver --fs_wal_dir=/var/lib/kudu/tserver --tserver_master_addrs=aserbin-kudu-jepsen-master.gce.cloudera.com:7051 --heap_profile_path=/tmp/kudu-tserver.1467 --unlock_experimental_flags=true --unlock_unsafe_flags=true --flagfile=/etc/kudu/conf/tserver.gflagfile --log_dir=/var/log/kudu --logtostderr=true Tablet server version: kudu 1.3.0-SNAPSHOT revision 3a23b058af0f4de04f8827255de405fff7bf9ee1 build type RELEASE built by jenkins at 09 Jan 2017 06:07:18 PST on aserbin-kudu-jepsen-control-node-debian8.gce.cloudera.com build id 2017-01-09_06-07-06 I0109 07:20:31.075892 1467 mem_tracker.cc:140] MemTracker: hard memory limit is 0.464404 GB I0109 07:20:31.075903 1467 mem_tracker.cc:142] MemTracker: soft memory limit is 0.278643 GB I0109 07:20:31.077199 1467 tablet_server_main.cc:63] Initializing tablet server... I0109 07:20:31.126713 1467 hybrid_clock.cc:177] HybridClock initialized. Resolution in nanos?: 1 Wait times tolerance adjustment: 1.0005 Current error: 463473 I0109 07:20:31.127518 1467 env_posix.cc:1286] Not raising process file limit of 65536; it is already as high as it can go I0109 07:20:31.127549 1467 file_cache.cc:401] Constructed file cache lbm with capacity 32768 I0109 07:20:31.128741 1467 fs_manager.cc:251] Opened local filesystem: /var/lib/kudu/tserver uuid: "1678276a36f849c28459de6b6788965a" format_stamp: "Formatted at 2017-01-09 15:20:10 on aserbin-kudu-jepsen-ts-5" I0109 07:20:31.132208 1467 ts_tablet_manager.cc:701] Loading metadata for tablet 9e99b710634a44f4a948132a5a180600 I0109 07:20:31.132675 1467 ts_tablet_manager.cc:863] Registered tablet 9e99b710634a44f4a948132a5a180600 I0109 07:20:31.132885 1481 ts_tablet_manager.cc:725] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a: Bootstrapping tablet I0109 07:20:31.133165 1481 tablet_bootstrap.cc:381] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a: Bootstrap starting. I0109 07:20:31.133529 1481 tablet_bootstrap.cc:540] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a: Time spent opening tablet: real 0.000s user 0.000s sys 0.000s I0109 07:20:31.133653 1481 tablet_bootstrap.cc:596] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a: Will attempt to recover log segment /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600/wal-000000001 to /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery/wal-000000001 I0109 07:20:31.133719 1481 tablet_bootstrap.cc:604] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a: Moving log directory /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600 to recovery directory /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery in preparation for log replay I0109 07:20:31.134001 1481 log_util.cc:315] Log segment /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery/wal-000000001 has no footer. This segment was likely being written when the server previously shut down. I0109 07:20:31.134035 1481 log_reader.cc:160] Log segment /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery/wal-000000001 was likely left in-progress after a previous crash. Will try to rebuild footer by scanning data. I0109 07:20:31.135290 1467 tablet_server_main.cc:66] Starting tablet server... I0109 07:20:31.138593 1481 log_util.cc:569] Scanning /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery/wal-000000001 for valid entry headers following offset 11219... I0109 07:20:31.246692 1481 log_util.cc:606] Found no log entry headers I0109 07:20:31.246765 1481 log_util.cc:218] Ignoring log segment corruption in /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery/wal-000000001 because there are no log entries following the corrupted one. The server probably crashed in the middle of writing an entry to the write-ahead log or downloaded an active log via tablet copy. Error detail: Corruption: CRC mismatch in log entry header: Log file corruption detected. Failed trying to read batch #0 at offset 11207 for log segment /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery/wal-000000001: Prior entries: [off=10942 REPLICATE (1.48)] [off=10976 COMMIT (1.48)] [off=11173 REPLICATE (1.49)] [off=11207 COMMIT (1.49)] I0109 07:20:31.246773 1481 log_util.cc:368] Successfully rebuilt footer for segment: /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery/wal-000000001 (valid entries through byte offset 11207) I0109 07:20:31.246866 1481 tablet.cc:983] T 9e99b710634a44f4a948132a5a180600 Rewinding schema during bootstrap to Schema [ 0:key[string NOT NULL], 1:value[int32 NOT NULL] ] I0109 07:20:31.247258 1481 log.cc:351] Log is configured to *not* fsync() on all Append() calls I0109 07:20:31.250198 1481 tablet_bootstrap.cc:381] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a: Bootstrap replayed 1/1 log segments. Stats: ops{read=49 overwritten=0 applied=49 ignored=0} inserts{seen=48 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I0109 07:20:31.250231 1481 tablet_bootstrap.cc:1017] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a: ReplayState: Previous OpId: 1.49, Committed OpId: 1.49, Pending Replicates: 0, Pending Commits: 0 I0109 07:20:31.250623 1481 tablet_bootstrap.cc:633] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a: Preparing to delete log recovery files and directory /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery I0109 07:20:31.250635 1481 tablet_bootstrap.cc:636] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a: Renaming log recovery dir from /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery to /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery-1483975231250634 I0109 07:20:31.250654 1481 tablet_bootstrap.cc:646] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a: Deleting all files from renamed log recovery directory /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery-1483975231250634 I0109 07:20:31.250766 1481 tablet_bootstrap.cc:649] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a: Completed deletion of old log recovery files and directory /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery-1483975231250634 I0109 07:20:31.250777 1481 tablet_bootstrap.cc:381] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a: Bootstrap complete. I0109 07:20:31.256198 1481 ts_tablet_manager.cc:730] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a: Time spent bootstrapping tablet: real 0.123s user 0.008s sys 0.040s I0109 07:20:31.256919 1481 raft_consensus.cc:287] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "710c236995db46699eb9055c19ac506e" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-4.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "1678276a36f849c28459de6b6788965a" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-5.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "591b1a1d383d4b659d7549653f4f136e" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-2.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "c5a3e756029444b2bf66ee0044f5bc35" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-1.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "7897578c54854ba0a5af32eb2aa04704" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-3.gce.cloudera.com" port: 7050 } } I0109 07:20:31.256949 1481 raft_consensus.cc:547] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1678276a36f849c28459de6b6788965a, State: 1, Role: FOLLOWER I0109 07:20:31.256963 1481 consensus_queue.cc:175] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 49, Last appended: 1.49, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER I0109 07:20:31.257153 1481 ts_tablet_manager.cc:757] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a: Time spent starting tablet: real 0.001s user 0.000s sys 0.004s I0109 07:20:31.263620 1467 rpc_server.cc:164] RPC server started. Bound to: 0.0.0.0:7050 I0109 07:20:31.263716 1467 webserver.cc:126] Starting webserver on 0.0.0.0:8050 I0109 07:20:31.263725 1467 webserver.cc:137] Document root disabled I0109 07:20:31.263880 1467 webserver.cc:221] Webserver started. Bound to: http://0.0.0.0:8050/ I0109 07:20:31.264492 1467 tablet_server_main.cc:69] Tablet server successfully started. I0109 07:20:31.277566 1554 raft_consensus.cc:2158] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 0.159s I0109 07:20:31.277629 1554 raft_consensus.cc:1744] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 710c236995db46699eb9055c19ac506e in term 1. I0109 07:20:31.316293 1597 heartbeater.cc:292] Connected to a master server at aserbin-kudu-jepsen-master.gce.cloudera.com:7051 I0109 07:20:31.316359 1597 heartbeater.cc:359] Registering TS with master... I0109 07:20:31.316488 1597 heartbeater.cc:373] Master aserbin-kudu-jepsen-master.gce.cloudera.com:7051 requested a full tablet report, sending... I0109 07:20:31.476933 1554 raft_consensus.cc:2202] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [term 1 FOLLOWER]: Advancing to term 2 I0109 07:20:31.477041 1554 raft_consensus.cc:2158] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [term 2 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 0.201s I0109 07:20:31.477497 1554 raft_consensus.cc:1744] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 710c236995db46699eb9055c19ac506e in term 2. I0109 07:20:31.842499 1529 raft_consensus.cc:410] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0109 07:20:31.842634 1529 raft_consensus.cc:2158] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [term 2 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 0.223s I0109 07:20:31.842713 1529 raft_consensus.cc:434] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "710c236995db46699eb9055c19ac506e" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-4.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "1678276a36f849c28459de6b6788965a" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-5.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "591b1a1d383d4b659d7549653f4f136e" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-2.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "c5a3e756029444b2bf66ee0044f5bc35" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-1.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "7897578c54854ba0a5af32eb2aa04704" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-3.gce.cloudera.com" port: 7050 } } I0109 07:20:32.041009 1554 raft_consensus.cc:1691] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 591b1a1d383d4b659d7549653f4f136e for term 3 because replica has last-logged OpId of term: 1 index: 49, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 46. I0109 07:20:32.041312 1529 leader_election.cc:215] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [CANDIDATE]: Term 3 pre-election: Requesting pre-vote from peer 710c236995db46699eb9055c19ac506e I0109 07:20:32.041466 1529 leader_election.cc:215] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [CANDIDATE]: Term 3 pre-election: Requesting pre-vote from peer 591b1a1d383d4b659d7549653f4f136e I0109 07:20:32.041584 1529 leader_election.cc:215] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [CANDIDATE]: Term 3 pre-election: Requesting pre-vote from peer c5a3e756029444b2bf66ee0044f5bc35 I0109 07:20:32.041707 1529 leader_election.cc:215] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [CANDIDATE]: Term 3 pre-election: Requesting pre-vote from peer 7897578c54854ba0a5af32eb2aa04704 I0109 07:20:32.043242 1479 leader_election.cc:347] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [CANDIDATE]: Term 3 pre-election: Vote granted by peer 591b1a1d383d4b659d7549653f4f136e I0109 07:20:32.073900 1554 raft_consensus.cc:2158] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [term 2 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 0.196s I0109 07:20:32.073961 1554 raft_consensus.cc:1744] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5a3e756029444b2bf66ee0044f5bc35 in term 2. I0109 07:20:32.103200 1476 leader_election.cc:361] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [CANDIDATE]: Term 3 pre-election: Vote denied by peer c5a3e756029444b2bf66ee0044f5bc35. Message: Service unavailable: T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1678276a36f849c28459de6b6788965a for term 3 because replica is already servicing an update from a current leader or another vote. I0109 07:20:32.173131 1479 leader_election.cc:361] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [CANDIDATE]: Term 3 pre-election: Vote denied by peer 7897578c54854ba0a5af32eb2aa04704. Message: Service unavailable: T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1678276a36f849c28459de6b6788965a for term 3 because replica is already servicing an update from a current leader or another vote. I0109 07:20:32.174243 1554 raft_consensus.cc:1691] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7897578c54854ba0a5af32eb2aa04704 for term 3 because replica has last-logged OpId of term: 1 index: 49, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 48. W0109 07:20:32.264544 1478 leader_election.cc:271] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 710c236995db46699eb9055c19ac506e: Timed out: RequestConsensusVote RPC to 172.31.114.38:7050 timed out after 0.223s (SENT) I0109 07:20:32.264621 1478 leader_election.cc:242] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. I0109 07:20:32.264930 1612 raft_consensus.cc:2158] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [term 2 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 0.195s I0109 07:20:32.264950 1612 raft_consensus.cc:1905] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority I0109 07:20:32.317641 1597 heartbeater.cc:365] Master aserbin-kudu-jepsen-master.gce.cloudera.com:7051 was elected leader, sending a full tablet report... I0109 07:20:32.626574 1529 raft_consensus.cc:410] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0109 07:20:32.626673 1529 raft_consensus.cc:2158] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [term 2 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 0.317s I0109 07:20:32.626749 1529 raft_consensus.cc:434] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "710c236995db46699eb9055c19ac506e" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-4.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "1678276a36f849c28459de6b6788965a" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-5.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "591b1a1d383d4b659d7549653f4f136e" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-2.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "c5a3e756029444b2bf66ee0044f5bc35" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-1.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "7897578c54854ba0a5af32eb2aa04704" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-3.gce.cloudera.com" port: 7050 } } I0109 07:20:32.824329 1554 raft_consensus.cc:2202] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [term 2 FOLLOWER]: Advancing to term 3 I0109 07:20:32.824434 1554 raft_consensus.cc:2158] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [term 3 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 0.250s I0109 07:20:32.824857 1554 raft_consensus.cc:1744] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5a3e756029444b2bf66ee0044f5bc35 in term 3. W0109 07:20:32.824959 1554 rpcz_store.cc:234] Call kudu.consensus.ConsensusService.RequestConsensusVote from 172.31.112.255:55111 (request call id 2) took 185ms (client timeout 164). W0109 07:20:32.825033 1554 rpcz_store.cc:238] Trace: 0109 07:20:32.638989 (+ 0us) service_pool.cc:143] Inserting onto call queue 0109 07:20:32.639014 (+ 25us) service_pool.cc:202] Handling call 0109 07:20:32.824932 (+185918us) spinlock_profiling.cc:255] Waited 185 ms on lock 0x45394e0. stack: 0000000001a20b21 00000000009d8b77 00000000007e48af 0000000000a8971f 0000000000a8a342 0000000001a2dc19 00007f73438340a3 00007f73420c487c ffffffffffffffff 0109 07:20:32.824944 (+ 12us) inbound_call.cc:130] Queueing success response Metrics: {"spinlock_wait_cycles":463160064} I0109 07:20:32.825183 1552 raft_consensus.cc:1674] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 591b1a1d383d4b659d7549653f4f136e in current term 3: Already voted for candidate c5a3e756029444b2bf66ee0044f5bc35 in this term. I0109 07:20:32.825351 1553 raft_consensus.cc:1719] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7897578c54854ba0a5af32eb2aa04704 for term 3 because replica is already servicing an update from a current leader or another vote. I0109 07:20:32.825531 1529 leader_election.cc:215] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [CANDIDATE]: Term 3 pre-election: Requesting pre-vote from peer 710c236995db46699eb9055c19ac506e I0109 07:20:32.825623 1529 leader_election.cc:215] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [CANDIDATE]: Term 3 pre-election: Requesting pre-vote from peer 591b1a1d383d4b659d7549653f4f136e I0109 07:20:32.825713 1529 leader_election.cc:215] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [CANDIDATE]: Term 3 pre-election: Requesting pre-vote from peer c5a3e756029444b2bf66ee0044f5bc35 I0109 07:20:32.825790 1529 leader_election.cc:215] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [CANDIDATE]: Term 3 pre-election: Requesting pre-vote from peer 7897578c54854ba0a5af32eb2aa04704 I0109 07:20:32.826078 1479 leader_election.cc:361] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [CANDIDATE]: Term 3 pre-election: Vote denied by peer 591b1a1d383d4b659d7549653f4f136e. Message: Invalid argument: T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1678276a36f849c28459de6b6788965a in current term 3: Already voted for candidate c5a3e756029444b2bf66ee0044f5bc35 in this term. I0109 07:20:32.826117 1479 leader_election.cc:361] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [CANDIDATE]: Term 3 pre-election: Vote denied by peer 7897578c54854ba0a5af32eb2aa04704. Message: Invalid argument: T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1678276a36f849c28459de6b6788965a in current term 3: Already voted for candidate c5a3e756029444b2bf66ee0044f5bc35 in this term. I0109 07:20:32.836022 1476 leader_election.cc:361] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [CANDIDATE]: Term 3 pre-election: Vote denied by peer c5a3e756029444b2bf66ee0044f5bc35. Message: Service unavailable: T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1678276a36f849c28459de6b6788965a for term 3 because replica is already servicing an update from a current leader or another vote. I0109 07:20:32.836097 1476 leader_election.cc:242] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. I0109 07:20:32.836228 1554 raft_consensus.cc:2158] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [term 3 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 0.292s I0109 07:20:32.836246 1554 raft_consensus.cc:1744] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5a3e756029444b2bf66ee0044f5bc35 in term 3. I0109 07:20:32.836496 1613 raft_consensus.cc:2158] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [term 3 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 0.319s I0109 07:20:32.836511 1613 raft_consensus.cc:1905] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority W0109 07:20:33.143622 1478 leader_election.cc:271] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 710c236995db46699eb9055c19ac506e: Timed out: RequestConsensusVote RPC to 172.31.114.38:7050 timed out after 0.317s (SENT) I0109 07:20:33.320358 1529 raft_consensus.cc:410] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0109 07:20:33.320441 1529 raft_consensus.cc:2158] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [term 3 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 0.400s I0109 07:20:33.320502 1529 raft_consensus.cc:434] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "710c236995db46699eb9055c19ac506e" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-4.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "1678276a36f849c28459de6b6788965a" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-5.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "591b1a1d383d4b659d7549653f4f136e" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-2.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "c5a3e756029444b2bf66ee0044f5bc35" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-1.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "7897578c54854ba0a5af32eb2aa04704" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-3.gce.cloudera.com" port: 7050 } } I0109 07:20:33.517554 1529 leader_election.cc:215] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [CANDIDATE]: Term 4 pre-election: Requesting pre-vote from peer 710c236995db46699eb9055c19ac506e I0109 07:20:33.517729 1529 leader_election.cc:215] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [CANDIDATE]: Term 4 pre-election: Requesting pre-vote from peer 591b1a1d383d4b659d7549653f4f136e I0109 07:20:33.517779 1529 leader_election.cc:215] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [CANDIDATE]: Term 4 pre-election: Requesting pre-vote from peer c5a3e756029444b2bf66ee0044f5bc35 I0109 07:20:33.517829 1529 leader_election.cc:215] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [CANDIDATE]: Term 4 pre-election: Requesting pre-vote from peer 7897578c54854ba0a5af32eb2aa04704 I0109 07:20:33.654139 1554 raft_consensus.cc:1691] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7897578c54854ba0a5af32eb2aa04704 for term 4 because replica has last-logged OpId of term: 1 index: 49, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 48. I0109 07:20:33.654296 1479 leader_election.cc:347] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [CANDIDATE]: Term 4 pre-election: Vote granted by peer 7897578c54854ba0a5af32eb2aa04704 I0109 07:20:33.666468 1479 leader_election.cc:347] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [CANDIDATE]: Term 4 pre-election: Vote granted by peer 591b1a1d383d4b659d7549653f4f136e I0109 07:20:33.666533 1479 leader_election.cc:242] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. I0109 07:20:33.666735 1554 raft_consensus.cc:1691] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 591b1a1d383d4b659d7549653f4f136e for term 4 because replica has last-logged OpId of term: 1 index: 49, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 46. I0109 07:20:33.666949 1614 raft_consensus.cc:2158] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [term 3 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 0.263s I0109 07:20:33.666966 1614 raft_consensus.cc:1968] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [term 3 FOLLOWER]: Leader pre-election won for term 4 I0109 07:20:33.666973 1614 raft_consensus.cc:410] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0109 07:20:33.666983 1614 raft_consensus.cc:2158] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [term 3 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 0.318s I0109 07:20:33.666990 1614 raft_consensus.cc:2202] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [term 3 FOLLOWER]: Advancing to term 4 I0109 07:20:33.667430 1614 raft_consensus.cc:434] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "710c236995db46699eb9055c19ac506e" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-4.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "1678276a36f849c28459de6b6788965a" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-5.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "591b1a1d383d4b659d7549653f4f136e" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-2.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "c5a3e756029444b2bf66ee0044f5bc35" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-1.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "7897578c54854ba0a5af32eb2aa04704" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-3.gce.cloudera.com" port: 7050 } } I0109 07:20:33.864277 1614 leader_election.cc:215] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [CANDIDATE]: Term 4 election: Requesting vote from peer 710c236995db46699eb9055c19ac506e I0109 07:20:33.864465 1614 leader_election.cc:215] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [CANDIDATE]: Term 4 election: Requesting vote from peer 591b1a1d383d4b659d7549653f4f136e I0109 07:20:33.864538 1614 leader_election.cc:215] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [CANDIDATE]: Term 4 election: Requesting vote from peer c5a3e756029444b2bf66ee0044f5bc35 I0109 07:20:33.864614 1614 leader_election.cc:215] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [CANDIDATE]: Term 4 election: Requesting vote from peer 7897578c54854ba0a5af32eb2aa04704 I0109 07:20:33.865495 1479 leader_election.cc:347] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [CANDIDATE]: Term 4 election: Vote granted by peer 591b1a1d383d4b659d7549653f4f136e I0109 07:20:33.865674 1479 leader_election.cc:347] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [CANDIDATE]: Term 4 election: Vote granted by peer 7897578c54854ba0a5af32eb2aa04704 I0109 07:20:33.865684 1479 leader_election.cc:242] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. I0109 07:20:33.865732 1614 raft_consensus.cc:2158] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [term 4 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 0.199s I0109 07:20:33.865779 1614 raft_consensus.cc:1968] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [term 4 FOLLOWER]: Leader election won for term 4 I0109 07:20:33.865811 1614 raft_consensus.cc:517] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [term 4 LEADER]: Becoming Leader. State: Replica: 1678276a36f849c28459de6b6788965a, State: 1, Role: LEADER I0109 07:20:33.865890 1614 consensus_queue.cc:157] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 49, Committed index: 49, Last appended: 1.49, Current term: 4, Majority size: 3, State: 1, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "710c236995db46699eb9055c19ac506e" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-4.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "1678276a36f849c28459de6b6788965a" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-5.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "591b1a1d383d4b659d7549653f4f136e" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-2.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "c5a3e756029444b2bf66ee0044f5bc35" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-1.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "7897578c54854ba0a5af32eb2aa04704" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-3.gce.cloudera.com" port: 7050 } } W0109 07:20:33.918684 1478 leader_election.cc:271] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 710c236995db46699eb9055c19ac506e: Timed out: RequestConsensusVote RPC to 172.31.114.38:7050 timed out after 0.400s (SENT) W0109 07:20:33.918764 1476 leader_election.cc:271] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer c5a3e756029444b2bf66ee0044f5bc35: Timed out: RequestConsensusVote RPC to 172.31.112.255:7050 timed out after 0.400s (SENT) I0109 07:20:34.106554 1614 consensus_queue.cc:693] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [LEADER]: Connected to new peer: Peer: 591b1a1d383d4b659d7549653f4f136e, Is new: false, Last received: 1.46, Next index: 47, Last known committed idx: 46, Last exchange result: ERROR, Needs tablet copy: false I0109 07:20:34.156733 1614 consensus_queue.cc:693] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [LEADER]: Connected to new peer: Peer: 7897578c54854ba0a5af32eb2aa04704, Is new: false, Last received: 1.48, Next index: 49, Last known committed idx: 48, Last exchange result: ERROR, Needs tablet copy: false W0109 07:20:34.183272 1478 leader_election.cc:271] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 710c236995db46699eb9055c19ac506e: Timed out: RequestConsensusVote RPC to 172.31.114.38:7050 timed out after 0.318s (SENT) W0109 07:20:34.183527 1476 leader_election.cc:271] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer c5a3e756029444b2bf66ee0044f5bc35: Timed out: RequestConsensusVote RPC to 172.31.112.255:7050 timed out after 0.318s (SENT) W0109 07:20:35.107054 1476 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a -> Peer c5a3e756029444b2bf66ee0044f5bc35 (aserbin-kudu-jepsen-ts-1.gce.cloudera.com:7050): Couldn't send request to peer c5a3e756029444b2bf66ee0044f5bc35 for tablet 9e99b710634a44f4a948132a5a180600. Status: Timed out: UpdateConsensus RPC to 172.31.112.255:7050 timed out after 1.000s (SENT). Retrying in the next heartbeat period. Already tried 1 times. W0109 07:20:35.107149 1478 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a -> Peer 710c236995db46699eb9055c19ac506e (aserbin-kudu-jepsen-ts-4.gce.cloudera.com:7050): Couldn't send request to peer 710c236995db46699eb9055c19ac506e for tablet 9e99b710634a44f4a948132a5a180600. Status: Timed out: UpdateConsensus RPC to 172.31.114.38:7050 timed out after 1.000s (SENT). Retrying in the next heartbeat period. Already tried 1 times. W0109 07:20:36.146894 1476 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a -> Peer c5a3e756029444b2bf66ee0044f5bc35 (aserbin-kudu-jepsen-ts-1.gce.cloudera.com:7050): Couldn't send request to peer c5a3e756029444b2bf66ee0044f5bc35 for tablet 9e99b710634a44f4a948132a5a180600. Status: Timed out: UpdateConsensus RPC to 172.31.112.255:7050 timed out after 1.000s (SENT). Retrying in the next heartbeat period. Already tried 2 times. W0109 07:20:36.156688 1478 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a -> Peer 710c236995db46699eb9055c19ac506e (aserbin-kudu-jepsen-ts-4.gce.cloudera.com:7050): Couldn't send request to peer 710c236995db46699eb9055c19ac506e for tablet 9e99b710634a44f4a948132a5a180600. Status: Timed out: UpdateConsensus RPC to 172.31.114.38:7050 timed out after 1.000s (SENT). Retrying in the next heartbeat period. Already tried 2 times. I0109 07:20:36.685220 1554 raft_consensus.cc:884] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [term 4 LEADER]: Rejecting Update request from peer 710c236995db46699eb9055c19ac506e for earlier term 2. Current term is 4. Ops: [] I0109 07:20:36.688436 1614 consensus_queue.cc:679] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [LEADER]: Peer 710c236995db46699eb9055c19ac506e log is divergent from this leader: its last log entry 2.50 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 49 I0109 07:20:36.688457 1614 consensus_queue.cc:693] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [LEADER]: Connected to new peer: Peer: 710c236995db46699eb9055c19ac506e, Is new: false, Last received: 0.0, Next index: 50, Last known committed idx: 49, Last exchange result: ERROR, Needs tablet copy: false W0109 07:20:37.196482 1476 consensus_peers.cc:356] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a -> Peer c5a3e756029444b2bf66ee0044f5bc35 (aserbin-kudu-jepsen-ts-1.gce.cloudera.com:7050): Couldn't send request to peer c5a3e756029444b2bf66ee0044f5bc35 for tablet 9e99b710634a44f4a948132a5a180600. Status: Timed out: UpdateConsensus RPC to 172.31.112.255:7050 timed out after 1.000s (SENT). Retrying in the next heartbeat period. Already tried 3 times. I0109 07:20:44.323132 1646 tablet_server_main.cc:56] Tablet server non-default flags: --raft_heartbeat_interval_ms=50 --fs_data_dirs=/var/lib/kudu/tserver --fs_wal_dir=/var/lib/kudu/tserver --tserver_master_addrs=aserbin-kudu-jepsen-master.gce.cloudera.com:7051 --heap_profile_path=/tmp/kudu-tserver.1646 --unlock_experimental_flags=true --unlock_unsafe_flags=true --flagfile=/etc/kudu/conf/tserver.gflagfile --log_dir=/var/log/kudu --logtostderr=true Tablet server version: kudu 1.3.0-SNAPSHOT revision 3a23b058af0f4de04f8827255de405fff7bf9ee1 build type RELEASE built by jenkins at 09 Jan 2017 06:07:18 PST on aserbin-kudu-jepsen-control-node-debian8.gce.cloudera.com build id 2017-01-09_06-07-06 I0109 07:20:44.323287 1646 mem_tracker.cc:140] MemTracker: hard memory limit is 0.464404 GB I0109 07:20:44.323297 1646 mem_tracker.cc:142] MemTracker: soft memory limit is 0.278643 GB I0109 07:20:44.324489 1646 tablet_server_main.cc:63] Initializing tablet server... I0109 07:20:44.376220 1646 hybrid_clock.cc:177] HybridClock initialized. Resolution in nanos?: 1 Wait times tolerance adjustment: 1.0005 Current error: 469973 I0109 07:20:44.377038 1646 env_posix.cc:1286] Not raising process file limit of 65536; it is already as high as it can go I0109 07:20:44.377073 1646 file_cache.cc:401] Constructed file cache lbm with capacity 32768 I0109 07:20:44.378147 1646 fs_manager.cc:251] Opened local filesystem: /var/lib/kudu/tserver uuid: "1678276a36f849c28459de6b6788965a" format_stamp: "Formatted at 2017-01-09 15:20:10 on aserbin-kudu-jepsen-ts-5" I0109 07:20:44.381387 1646 ts_tablet_manager.cc:701] Loading metadata for tablet 9e99b710634a44f4a948132a5a180600 I0109 07:20:44.381647 1646 ts_tablet_manager.cc:863] Registered tablet 9e99b710634a44f4a948132a5a180600 I0109 07:20:44.381867 1660 ts_tablet_manager.cc:725] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a: Bootstrapping tablet I0109 07:20:44.382103 1660 tablet_bootstrap.cc:381] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a: Bootstrap starting. I0109 07:20:44.382381 1660 tablet_bootstrap.cc:540] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a: Time spent opening tablet: real 0.000s user 0.000s sys 0.000s I0109 07:20:44.382510 1660 tablet_bootstrap.cc:596] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a: Will attempt to recover log segment /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600/wal-000000001 to /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery/wal-000000001 I0109 07:20:44.382547 1660 tablet_bootstrap.cc:604] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a: Moving log directory /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600 to recovery directory /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery in preparation for log replay I0109 07:20:44.382748 1660 log_util.cc:315] Log segment /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery/wal-000000001 has no footer. This segment was likely being written when the server previously shut down. I0109 07:20:44.382763 1660 log_reader.cc:160] Log segment /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery/wal-000000001 was likely left in-progress after a previous crash. Will try to rebuild footer by scanning data. I0109 07:20:44.384220 1646 tablet_server_main.cc:66] Starting tablet server... I0109 07:20:44.386205 1660 log_util.cc:569] Scanning /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery/wal-000000001 for valid entry headers following offset 13824... I0109 07:20:44.471035 1660 log_util.cc:606] Found no log entry headers I0109 07:20:44.471099 1660 log_util.cc:218] Ignoring log segment corruption in /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery/wal-000000001 because there are no log entries following the corrupted one. The server probably crashed in the middle of writing an entry to the write-ahead log or downloaded an active log via tablet copy. Error detail: Corruption: CRC mismatch in log entry header: Log file corruption detected. Failed trying to read batch #0 at offset 13812 for log segment /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery/wal-000000001: Prior entries: [off=13547 REPLICATE (4.60)] [off=13581 COMMIT (4.60)] [off=13778 REPLICATE (4.61)] [off=13812 COMMIT (4.61)] I0109 07:20:44.471109 1660 log_util.cc:368] Successfully rebuilt footer for segment: /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery/wal-000000001 (valid entries through byte offset 13812) I0109 07:20:44.471165 1660 tablet.cc:983] T 9e99b710634a44f4a948132a5a180600 Rewinding schema during bootstrap to Schema [ 0:key[string NOT NULL], 1:value[int32 NOT NULL] ] I0109 07:20:44.471385 1660 log.cc:351] Log is configured to *not* fsync() on all Append() calls I0109 07:20:44.474395 1660 tablet_bootstrap.cc:381] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a: Bootstrap replayed 1/1 log segments. Stats: ops{read=61 overwritten=0 applied=61 ignored=0} inserts{seen=59 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I0109 07:20:44.474422 1660 tablet_bootstrap.cc:1017] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a: ReplayState: Previous OpId: 4.61, Committed OpId: 4.61, Pending Replicates: 0, Pending Commits: 0 I0109 07:20:44.474779 1660 tablet_bootstrap.cc:633] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a: Preparing to delete log recovery files and directory /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery I0109 07:20:44.474791 1660 tablet_bootstrap.cc:636] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a: Renaming log recovery dir from /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery to /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery-1483975244474790 I0109 07:20:44.474808 1660 tablet_bootstrap.cc:646] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a: Deleting all files from renamed log recovery directory /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery-1483975244474790 I0109 07:20:44.474900 1660 tablet_bootstrap.cc:649] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a: Completed deletion of old log recovery files and directory /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery-1483975244474790 I0109 07:20:44.474917 1660 tablet_bootstrap.cc:381] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a: Bootstrap complete. I0109 07:20:44.480329 1660 ts_tablet_manager.cc:730] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a: Time spent bootstrapping tablet: real 0.098s user 0.012s sys 0.036s I0109 07:20:44.480978 1660 raft_consensus.cc:287] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [term 4 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "710c236995db46699eb9055c19ac506e" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-4.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "1678276a36f849c28459de6b6788965a" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-5.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "591b1a1d383d4b659d7549653f4f136e" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-2.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "c5a3e756029444b2bf66ee0044f5bc35" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-1.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "7897578c54854ba0a5af32eb2aa04704" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-3.gce.cloudera.com" port: 7050 } } I0109 07:20:44.481021 1660 raft_consensus.cc:547] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [term 4 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1678276a36f849c28459de6b6788965a, State: 1, Role: FOLLOWER I0109 07:20:44.481040 1660 consensus_queue.cc:175] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 61, Last appended: 4.61, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER I0109 07:20:44.481253 1660 ts_tablet_manager.cc:757] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s I0109 07:20:44.488565 1646 rpc_server.cc:164] RPC server started. Bound to: 0.0.0.0:7050 I0109 07:20:44.488658 1646 webserver.cc:126] Starting webserver on 0.0.0.0:8050 I0109 07:20:44.488665 1646 webserver.cc:137] Document root disabled I0109 07:20:44.488817 1646 webserver.cc:221] Webserver started. Bound to: http://0.0.0.0:8050/ I0109 07:20:44.489264 1646 tablet_server_main.cc:69] Tablet server successfully started. I0109 07:20:44.546859 1777 heartbeater.cc:292] Connected to a master server at aserbin-kudu-jepsen-master.gce.cloudera.com:7051 I0109 07:20:44.546926 1777 heartbeater.cc:359] Registering TS with master... I0109 07:20:44.547041 1777 heartbeater.cc:373] Master aserbin-kudu-jepsen-master.gce.cloudera.com:7051 requested a full tablet report, sending... I0109 07:20:44.645334 1733 raft_consensus.cc:2158] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [term 4 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 0.168s I0109 07:20:44.645422 1733 raft_consensus.cc:1744] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 710c236995db46699eb9055c19ac506e in term 4. I0109 07:20:44.706311 1733 raft_consensus.cc:2158] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [term 4 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 0.224s I0109 07:20:44.706370 1733 raft_consensus.cc:1744] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 591b1a1d383d4b659d7549653f4f136e in term 4. I0109 07:20:44.737867 1733 raft_consensus.cc:1691] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c5a3e756029444b2bf66ee0044f5bc35 for term 5 because replica has last-logged OpId of term: 4 index: 61, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 49. I0109 07:20:44.903600 1733 raft_consensus.cc:2202] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [term 4 FOLLOWER]: Advancing to term 5 I0109 07:20:44.903708 1733 raft_consensus.cc:2158] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [term 5 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 0.171s I0109 07:20:44.904204 1733 raft_consensus.cc:1744] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 710c236995db46699eb9055c19ac506e in term 5. I0109 07:20:44.904342 1733 raft_consensus.cc:1674] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 591b1a1d383d4b659d7549653f4f136e in current term 5: Already voted for candidate 710c236995db46699eb9055c19ac506e in this term. I0109 07:20:45.103674 1733 raft_consensus.cc:915] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [term 5 FOLLOWER]: Refusing update from remote peer 710c236995db46699eb9055c19ac506e: Log matching property violated. Preceding OpId in replica: term: 4 index: 61. Preceding OpId from leader: term: 5 index: 62. (index mismatch) I0109 07:20:45.104621 1777 heartbeater.cc:365] Master aserbin-kudu-jepsen-master.gce.cloudera.com:7051 was elected leader, sending a full tablet report... I0109 07:21:00.897737 1702 raft_consensus.cc:410] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 710c236995db46699eb9055c19ac506e) I0109 07:21:00.897850 1702 raft_consensus.cc:2158] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [term 5 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 0.207s I0109 07:21:00.897931 1702 raft_consensus.cc:434] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "710c236995db46699eb9055c19ac506e" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-4.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "1678276a36f849c28459de6b6788965a" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-5.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "591b1a1d383d4b659d7549653f4f136e" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-2.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "c5a3e756029444b2bf66ee0044f5bc35" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-1.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "7897578c54854ba0a5af32eb2aa04704" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-3.gce.cloudera.com" port: 7050 } } I0109 07:21:07.105178 1810 tablet_server_main.cc:56] Tablet server non-default flags: --raft_heartbeat_interval_ms=50 --fs_data_dirs=/var/lib/kudu/tserver --fs_wal_dir=/var/lib/kudu/tserver --tserver_master_addrs=aserbin-kudu-jepsen-master.gce.cloudera.com:7051 --heap_profile_path=/tmp/kudu-tserver.1810 --unlock_experimental_flags=true --unlock_unsafe_flags=true --flagfile=/etc/kudu/conf/tserver.gflagfile --log_dir=/var/log/kudu --logtostderr=true Tablet server version: kudu 1.3.0-SNAPSHOT revision 3a23b058af0f4de04f8827255de405fff7bf9ee1 build type RELEASE built by jenkins at 09 Jan 2017 06:07:18 PST on aserbin-kudu-jepsen-control-node-debian8.gce.cloudera.com build id 2017-01-09_06-07-06 I0109 07:21:07.105375 1810 mem_tracker.cc:140] MemTracker: hard memory limit is 0.464404 GB I0109 07:21:07.105386 1810 mem_tracker.cc:142] MemTracker: soft memory limit is 0.278643 GB I0109 07:21:07.106604 1810 tablet_server_main.cc:63] Initializing tablet server... I0109 07:21:07.156605 1810 hybrid_clock.cc:177] HybridClock initialized. Resolution in nanos?: 1 Wait times tolerance adjustment: 1.0005 Current error: 481473 I0109 07:21:07.157424 1810 env_posix.cc:1286] Not raising process file limit of 65536; it is already as high as it can go I0109 07:21:07.157462 1810 file_cache.cc:401] Constructed file cache lbm with capacity 32768 I0109 07:21:07.158583 1810 fs_manager.cc:251] Opened local filesystem: /var/lib/kudu/tserver uuid: "1678276a36f849c28459de6b6788965a" format_stamp: "Formatted at 2017-01-09 15:20:10 on aserbin-kudu-jepsen-ts-5" I0109 07:21:07.162179 1810 ts_tablet_manager.cc:701] Loading metadata for tablet 9e99b710634a44f4a948132a5a180600 I0109 07:21:07.162502 1810 ts_tablet_manager.cc:863] Registered tablet 9e99b710634a44f4a948132a5a180600 I0109 07:21:07.162684 1824 ts_tablet_manager.cc:725] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a: Bootstrapping tablet I0109 07:21:07.162904 1824 tablet_bootstrap.cc:381] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a: Bootstrap starting. I0109 07:21:07.163174 1824 tablet_bootstrap.cc:540] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a: Time spent opening tablet: real 0.000s user 0.000s sys 0.000s I0109 07:21:07.163251 1824 tablet_bootstrap.cc:596] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a: Will attempt to recover log segment /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600/wal-000000001 to /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery/wal-000000001 I0109 07:21:07.163264 1824 tablet_bootstrap.cc:604] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a: Moving log directory /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600 to recovery directory /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery in preparation for log replay I0109 07:21:07.163508 1824 log_util.cc:315] Log segment /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery/wal-000000001 has no footer. This segment was likely being written when the server previously shut down. I0109 07:21:07.163522 1824 log_reader.cc:160] Log segment /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery/wal-000000001 was likely left in-progress after a previous crash. Will try to rebuild footer by scanning data. I0109 07:21:07.164984 1810 tablet_server_main.cc:66] Starting tablet server... I0109 07:21:07.167106 1824 log_util.cc:569] Scanning /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery/wal-000000001 for valid entry headers following offset 36790... I0109 07:21:07.242957 1824 log_util.cc:606] Found no log entry headers I0109 07:21:07.243203 1824 log_util.cc:218] Ignoring log segment corruption in /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery/wal-000000001 because there are no log entries following the corrupted one. The server probably crashed in the middle of writing an entry to the write-ahead log or downloaded an active log via tablet copy. Error detail: Corruption: CRC mismatch in log entry header: Log file corruption detected. Failed trying to read batch #0 at offset 36778 for log segment /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery/wal-000000001: Prior entries: [off=36347 COMMIT (5.159)] [off=36545 REPLICATE (5.160)] [off=36580 COMMIT (5.160)] [off=36778 REPLICATE (5.161)] I0109 07:21:07.243271 1824 log_util.cc:368] Successfully rebuilt footer for segment: /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery/wal-000000001 (valid entries through byte offset 36778) I0109 07:21:07.243381 1824 tablet.cc:983] T 9e99b710634a44f4a948132a5a180600 Rewinding schema during bootstrap to Schema [ 0:key[string NOT NULL], 1:value[int32 NOT NULL] ] I0109 07:21:07.243861 1824 log.cc:351] Log is configured to *not* fsync() on all Append() calls I0109 07:21:07.257053 1824 tablet_bootstrap.cc:381] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a: Bootstrap replayed 1/1 log segments. Stats: ops{read=161 overwritten=0 applied=160 ignored=0} inserts{seen=157 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates I0109 07:21:07.257256 1824 tablet_bootstrap.cc:1017] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a: ReplayState: Previous OpId: 5.161, Committed OpId: 5.160, Pending Replicates: 1, Pending Commits: 0 I0109 07:21:07.257277 1824 tablet_bootstrap.cc:1017] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a: Dumping REPLICATES: I0109 07:21:07.257282 1824 tablet_bootstrap.cc:1017] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a: type: REPLICATE replicate { id { term: 5 index: 161 } timestamp: 6078362667140030464 op_type: WRITE_OP write_request { tablet_id: "9e99b710634a44f4a948132a5a180600" schema { columns { name: "key" type: STRING is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "value" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } row_operations { rows: "\005\003\000\000\000\000\000\000\000\000\001\000\000\000\000\000\000\000\007\000\000\000" indirect_data: "x" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 6078362665418371072 } request_id { client_id: "a89de875ce2044048caa38631f674301" seq_no: 32 first_incomplete_seq_no: 32 attempt_no: 1 } } I0109 07:21:07.257803 1824 tablet_bootstrap.cc:633] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a: Preparing to delete log recovery files and directory /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery I0109 07:21:07.257815 1824 tablet_bootstrap.cc:636] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a: Renaming log recovery dir from /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery to /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery-1483975267257814 I0109 07:21:07.257849 1824 tablet_bootstrap.cc:646] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a: Deleting all files from renamed log recovery directory /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery-1483975267257814 I0109 07:21:07.257985 1824 tablet_bootstrap.cc:649] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a: Completed deletion of old log recovery files and directory /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery-1483975267257814 I0109 07:21:07.257997 1824 tablet_bootstrap.cc:381] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a: Bootstrap complete. I0109 07:21:07.263597 1824 ts_tablet_manager.cc:730] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a: Time spent bootstrapping tablet: real 0.101s user 0.012s sys 0.040s I0109 07:21:07.264369 1824 raft_consensus.cc:287] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [term 5 FOLLOWER]: Replica starting. Triggering 1 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "710c236995db46699eb9055c19ac506e" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-4.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "1678276a36f849c28459de6b6788965a" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-5.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "591b1a1d383d4b659d7549653f4f136e" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-2.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "c5a3e756029444b2bf66ee0044f5bc35" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-1.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "7897578c54854ba0a5af32eb2aa04704" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-3.gce.cloudera.com" port: 7050 } } I0109 07:21:07.264672 1824 raft_consensus.cc:547] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [term 5 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1678276a36f849c28459de6b6788965a, State: 1, Role: FOLLOWER I0109 07:21:07.264698 1824 consensus_queue.cc:175] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 160, Last appended: 5.161, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER I0109 07:21:07.264964 1824 ts_tablet_manager.cc:757] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s I0109 07:21:07.271338 1810 rpc_server.cc:164] RPC server started. Bound to: 0.0.0.0:7050 I0109 07:21:07.271447 1810 webserver.cc:126] Starting webserver on 0.0.0.0:8050 I0109 07:21:07.271461 1810 webserver.cc:137] Document root disabled I0109 07:21:07.271613 1810 webserver.cc:221] Webserver started. Bound to: http://0.0.0.0:8050/ I0109 07:21:07.272151 1810 tablet_server_main.cc:69] Tablet server successfully started. I0109 07:21:07.323909 1941 heartbeater.cc:292] Connected to a master server at aserbin-kudu-jepsen-master.gce.cloudera.com:7051 I0109 07:21:07.324012 1941 heartbeater.cc:359] Registering TS with master... I0109 07:21:07.324148 1941 heartbeater.cc:373] Master aserbin-kudu-jepsen-master.gce.cloudera.com:7051 requested a full tablet report, sending... I0109 07:21:07.374918 1898 raft_consensus.cc:2202] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [term 5 FOLLOWER]: Advancing to term 6 I0109 07:21:07.375001 1898 raft_consensus.cc:2158] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [term 6 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 0.186s I0109 07:21:07.375406 1898 raft_consensus.cc:1744] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 710c236995db46699eb9055c19ac506e in term 6. I0109 07:21:07.572670 1898 raft_consensus.cc:915] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [term 6 FOLLOWER]: Refusing update from remote peer 710c236995db46699eb9055c19ac506e: Log matching property violated. Preceding OpId in replica: term: 5 index: 161. Preceding OpId from leader: term: 6 index: 162. (index mismatch) I0109 07:21:07.573879 1941 heartbeater.cc:365] Master aserbin-kudu-jepsen-master.gce.cloudera.com:7051 was elected leader, sending a full tablet report... I0109 07:21:20.190520 1984 tablet_server_main.cc:56] Tablet server non-default flags: --raft_heartbeat_interval_ms=50 --fs_data_dirs=/var/lib/kudu/tserver --fs_wal_dir=/var/lib/kudu/tserver --tserver_master_addrs=aserbin-kudu-jepsen-master.gce.cloudera.com:7051 --heap_profile_path=/tmp/kudu-tserver.1984 --unlock_experimental_flags=true --unlock_unsafe_flags=true --flagfile=/etc/kudu/conf/tserver.gflagfile --log_dir=/var/log/kudu --logtostderr=true Tablet server version: kudu 1.3.0-SNAPSHOT revision 3a23b058af0f4de04f8827255de405fff7bf9ee1 build type RELEASE built by jenkins at 09 Jan 2017 06:07:18 PST on aserbin-kudu-jepsen-control-node-debian8.gce.cloudera.com build id 2017-01-09_06-07-06 I0109 07:21:20.190742 1984 mem_tracker.cc:140] MemTracker: hard memory limit is 0.464404 GB I0109 07:21:20.190754 1984 mem_tracker.cc:142] MemTracker: soft memory limit is 0.278643 GB I0109 07:21:20.191983 1984 tablet_server_main.cc:63] Initializing tablet server... I0109 07:21:20.241761 1984 hybrid_clock.cc:177] HybridClock initialized. Resolution in nanos?: 1 Wait times tolerance adjustment: 1.0005 Current error: 487973 I0109 07:21:20.242570 1984 env_posix.cc:1286] Not raising process file limit of 65536; it is already as high as it can go I0109 07:21:20.242600 1984 file_cache.cc:401] Constructed file cache lbm with capacity 32768 I0109 07:21:20.243602 1984 fs_manager.cc:251] Opened local filesystem: /var/lib/kudu/tserver uuid: "1678276a36f849c28459de6b6788965a" format_stamp: "Formatted at 2017-01-09 15:20:10 on aserbin-kudu-jepsen-ts-5" I0109 07:21:20.246978 1984 ts_tablet_manager.cc:701] Loading metadata for tablet 9e99b710634a44f4a948132a5a180600 I0109 07:21:20.247248 1984 ts_tablet_manager.cc:863] Registered tablet 9e99b710634a44f4a948132a5a180600 I0109 07:21:20.247418 1998 ts_tablet_manager.cc:725] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a: Bootstrapping tablet I0109 07:21:20.247613 1998 tablet_bootstrap.cc:381] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a: Bootstrap starting. I0109 07:21:20.247856 1998 tablet_bootstrap.cc:540] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a: Time spent opening tablet: real 0.000s user 0.000s sys 0.000s I0109 07:21:20.247973 1998 tablet_bootstrap.cc:596] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a: Will attempt to recover log segment /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600/wal-000000001 to /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery/wal-000000001 I0109 07:21:20.247988 1998 tablet_bootstrap.cc:604] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a: Moving log directory /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600 to recovery directory /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery in preparation for log replay I0109 07:21:20.248159 1998 log_util.cc:315] Log segment /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery/wal-000000001 has no footer. This segment was likely being written when the server previously shut down. I0109 07:21:20.248175 1998 log_reader.cc:160] Log segment /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery/wal-000000001 was likely left in-progress after a previous crash. Will try to rebuild footer by scanning data. I0109 07:21:20.249732 1984 tablet_server_main.cc:66] Starting tablet server... I0109 07:21:20.251902 1998 log_util.cc:569] Scanning /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery/wal-000000001 for valid entry headers following offset 47842... I0109 07:21:20.324635 1998 log_util.cc:606] Found no log entry headers I0109 07:21:20.324712 1998 log_util.cc:218] Ignoring log segment corruption in /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery/wal-000000001 because there are no log entries following the corrupted one. The server probably crashed in the middle of writing an entry to the write-ahead log or downloaded an active log via tablet copy. Error detail: Corruption: CRC mismatch in log entry header: Log file corruption detected. Failed trying to read batch #0 at offset 47830 for log segment /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery/wal-000000001: Prior entries: [off=47562 REPLICATE (6.208)] [off=47597 COMMIT (6.208)] [off=47795 REPLICATE (6.209)] [off=47830 COMMIT (6.209)] I0109 07:21:20.324723 1998 log_util.cc:368] Successfully rebuilt footer for segment: /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery/wal-000000001 (valid entries through byte offset 47830) I0109 07:21:20.324818 1998 tablet.cc:983] T 9e99b710634a44f4a948132a5a180600 Rewinding schema during bootstrap to Schema [ 0:key[string NOT NULL], 1:value[int32 NOT NULL] ] I0109 07:21:20.325086 1998 log.cc:351] Log is configured to *not* fsync() on all Append() calls I0109 07:21:20.343417 1998 tablet_bootstrap.cc:381] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a: Bootstrap replayed 1/1 log segments. Stats: ops{read=209 overwritten=0 applied=209 ignored=0} inserts{seen=205 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I0109 07:21:20.343478 1998 tablet_bootstrap.cc:1017] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a: ReplayState: Previous OpId: 6.209, Committed OpId: 6.209, Pending Replicates: 0, Pending Commits: 0 I0109 07:21:20.344255 1998 tablet_bootstrap.cc:633] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a: Preparing to delete log recovery files and directory /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery I0109 07:21:20.344292 1998 tablet_bootstrap.cc:636] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a: Renaming log recovery dir from /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery to /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery-1483975280344289 I0109 07:21:20.344316 1998 tablet_bootstrap.cc:646] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a: Deleting all files from renamed log recovery directory /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery-1483975280344289 I0109 07:21:20.344488 1998 tablet_bootstrap.cc:649] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a: Completed deletion of old log recovery files and directory /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery-1483975280344289 I0109 07:21:20.344501 1998 tablet_bootstrap.cc:381] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a: Bootstrap complete. I0109 07:21:20.350121 1998 ts_tablet_manager.cc:730] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a: Time spent bootstrapping tablet: real 0.103s user 0.020s sys 0.036s I0109 07:21:20.350896 1998 raft_consensus.cc:287] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [term 6 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "710c236995db46699eb9055c19ac506e" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-4.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "1678276a36f849c28459de6b6788965a" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-5.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "591b1a1d383d4b659d7549653f4f136e" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-2.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "c5a3e756029444b2bf66ee0044f5bc35" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-1.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "7897578c54854ba0a5af32eb2aa04704" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-3.gce.cloudera.com" port: 7050 } } I0109 07:21:20.350924 1998 raft_consensus.cc:547] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [term 6 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1678276a36f849c28459de6b6788965a, State: 1, Role: FOLLOWER I0109 07:21:20.350944 1998 consensus_queue.cc:175] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 209, Last appended: 6.209, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER I0109 07:21:20.351135 1998 ts_tablet_manager.cc:757] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s I0109 07:21:20.358042 1984 rpc_server.cc:164] RPC server started. Bound to: 0.0.0.0:7050 I0109 07:21:20.358139 1984 webserver.cc:126] Starting webserver on 0.0.0.0:8050 I0109 07:21:20.358145 1984 webserver.cc:137] Document root disabled I0109 07:21:20.358301 1984 webserver.cc:221] Webserver started. Bound to: http://0.0.0.0:8050/ I0109 07:21:20.358880 1984 tablet_server_main.cc:69] Tablet server successfully started. I0109 07:21:20.411166 2114 heartbeater.cc:292] Connected to a master server at aserbin-kudu-jepsen-master.gce.cloudera.com:7051 I0109 07:21:20.411237 2114 heartbeater.cc:359] Registering TS with master... I0109 07:21:20.411367 2114 heartbeater.cc:373] Master aserbin-kudu-jepsen-master.gce.cloudera.com:7051 requested a full tablet report, sending... I0109 07:21:20.492606 2071 raft_consensus.cc:1691] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7897578c54854ba0a5af32eb2aa04704 for term 7 because replica has last-logged OpId of term: 6 index: 209, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 206. I0109 07:21:20.520481 2046 raft_consensus.cc:410] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0109 07:21:20.520608 2046 raft_consensus.cc:2158] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [term 6 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 0.193s I0109 07:21:20.520687 2046 raft_consensus.cc:434] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "710c236995db46699eb9055c19ac506e" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-4.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "1678276a36f849c28459de6b6788965a" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-5.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "591b1a1d383d4b659d7549653f4f136e" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-2.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "c5a3e756029444b2bf66ee0044f5bc35" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-1.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "7897578c54854ba0a5af32eb2aa04704" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-3.gce.cloudera.com" port: 7050 } } I0109 07:21:20.717419 2046 leader_election.cc:215] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [CANDIDATE]: Term 7 pre-election: Requesting pre-vote from peer 710c236995db46699eb9055c19ac506e I0109 07:21:20.717638 2046 leader_election.cc:215] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [CANDIDATE]: Term 7 pre-election: Requesting pre-vote from peer 591b1a1d383d4b659d7549653f4f136e I0109 07:21:20.717773 2046 leader_election.cc:215] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [CANDIDATE]: Term 7 pre-election: Requesting pre-vote from peer c5a3e756029444b2bf66ee0044f5bc35 I0109 07:21:20.717861 2046 leader_election.cc:215] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [CANDIDATE]: Term 7 pre-election: Requesting pre-vote from peer 7897578c54854ba0a5af32eb2aa04704 W0109 07:21:20.718777 1995 leader_election.cc:271] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 710c236995db46699eb9055c19ac506e: Network error: Client connection negotiation failed: client connection to 172.31.114.38:7050: connect: Connection refused (error 111) W0109 07:21:20.718875 1996 leader_election.cc:271] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 591b1a1d383d4b659d7549653f4f136e: Network error: Client connection negotiation failed: client connection to 172.31.113.2:7050: connect: Connection refused (error 111) W0109 07:21:20.718943 1993 leader_election.cc:271] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer c5a3e756029444b2bf66ee0044f5bc35: Network error: Client connection negotiation failed: client connection to 172.31.112.255:7050: connect: Connection refused (error 111) I0109 07:21:20.718955 1993 leader_election.cc:242] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. I0109 07:21:20.719238 2120 raft_consensus.cc:2158] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [term 6 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 0.201s I0109 07:21:20.719344 2120 raft_consensus.cc:1905] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority I0109 07:21:20.719661 1996 leader_election.cc:347] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [CANDIDATE]: Term 7 pre-election: Vote granted by peer 7897578c54854ba0a5af32eb2aa04704 I0109 07:21:21.099691 2046 raft_consensus.cc:410] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0109 07:21:21.099758 2046 raft_consensus.cc:2158] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [term 6 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 0.239s I0109 07:21:21.099822 2046 raft_consensus.cc:434] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "710c236995db46699eb9055c19ac506e" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-4.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "1678276a36f849c28459de6b6788965a" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-5.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "591b1a1d383d4b659d7549653f4f136e" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-2.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "c5a3e756029444b2bf66ee0044f5bc35" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-1.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "7897578c54854ba0a5af32eb2aa04704" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-3.gce.cloudera.com" port: 7050 } } I0109 07:21:21.296902 2046 leader_election.cc:215] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [CANDIDATE]: Term 7 pre-election: Requesting pre-vote from peer 710c236995db46699eb9055c19ac506e I0109 07:21:21.297219 2046 leader_election.cc:215] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [CANDIDATE]: Term 7 pre-election: Requesting pre-vote from peer 591b1a1d383d4b659d7549653f4f136e I0109 07:21:21.297340 2046 leader_election.cc:215] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [CANDIDATE]: Term 7 pre-election: Requesting pre-vote from peer c5a3e756029444b2bf66ee0044f5bc35 W0109 07:21:21.297412 1995 leader_election.cc:271] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 710c236995db46699eb9055c19ac506e: Network error: Client connection negotiation failed: client connection to 172.31.114.38:7050: connect: Connection refused (error 111) I0109 07:21:21.297530 2046 leader_election.cc:215] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [CANDIDATE]: Term 7 pre-election: Requesting pre-vote from peer 7897578c54854ba0a5af32eb2aa04704 W0109 07:21:21.297890 1993 leader_election.cc:271] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer c5a3e756029444b2bf66ee0044f5bc35: Network error: Client connection negotiation failed: client connection to 172.31.112.255:7050: connect: Connection refused (error 111) W0109 07:21:21.297943 1996 leader_election.cc:271] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 591b1a1d383d4b659d7549653f4f136e: Network error: Client connection negotiation failed: client connection to 172.31.113.2:7050: connect: Connection refused (error 111) I0109 07:21:21.297955 1996 leader_election.cc:242] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. I0109 07:21:21.298101 2130 raft_consensus.cc:2158] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [term 6 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 0.170s I0109 07:21:21.298118 2130 raft_consensus.cc:1905] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority I0109 07:21:21.326989 1996 leader_election.cc:347] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [CANDIDATE]: Term 7 pre-election: Vote granted by peer 7897578c54854ba0a5af32eb2aa04704 I0109 07:21:21.327270 2071 raft_consensus.cc:1691] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7897578c54854ba0a5af32eb2aa04704 for term 7 because replica has last-logged OpId of term: 6 index: 209, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 206. I0109 07:21:21.412423 2114 heartbeater.cc:365] Master aserbin-kudu-jepsen-master.gce.cloudera.com:7051 was elected leader, sending a full tablet report... I0109 07:21:21.671340 2046 raft_consensus.cc:410] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0109 07:21:21.671418 2046 raft_consensus.cc:2158] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [term 6 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 0.213s I0109 07:21:21.671489 2046 raft_consensus.cc:434] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "710c236995db46699eb9055c19ac506e" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-4.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "1678276a36f849c28459de6b6788965a" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-5.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "591b1a1d383d4b659d7549653f4f136e" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-2.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "c5a3e756029444b2bf66ee0044f5bc35" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-1.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "7897578c54854ba0a5af32eb2aa04704" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-3.gce.cloudera.com" port: 7050 } } I0109 07:21:21.868115 2046 leader_election.cc:215] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [CANDIDATE]: Term 7 pre-election: Requesting pre-vote from peer 710c236995db46699eb9055c19ac506e I0109 07:21:21.868361 2046 leader_election.cc:215] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [CANDIDATE]: Term 7 pre-election: Requesting pre-vote from peer 591b1a1d383d4b659d7549653f4f136e I0109 07:21:21.868464 2046 leader_election.cc:215] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [CANDIDATE]: Term 7 pre-election: Requesting pre-vote from peer c5a3e756029444b2bf66ee0044f5bc35 I0109 07:21:21.868594 2046 leader_election.cc:215] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [CANDIDATE]: Term 7 pre-election: Requesting pre-vote from peer 7897578c54854ba0a5af32eb2aa04704 W0109 07:21:21.869282 1995 leader_election.cc:271] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 710c236995db46699eb9055c19ac506e: Network error: Client connection negotiation failed: client connection to 172.31.114.38:7050: connect: Connection refused (error 111) W0109 07:21:21.869328 1996 leader_election.cc:271] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 591b1a1d383d4b659d7549653f4f136e: Network error: Client connection negotiation failed: client connection to 172.31.113.2:7050: connect: Connection refused (error 111) W0109 07:21:21.869356 1993 leader_election.cc:271] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer c5a3e756029444b2bf66ee0044f5bc35: Network error: Client connection negotiation failed: client connection to 172.31.112.255:7050: connect: Connection refused (error 111) I0109 07:21:21.869365 1993 leader_election.cc:242] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. I0109 07:21:21.869518 2134 raft_consensus.cc:2158] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [term 6 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 0.241s I0109 07:21:21.869534 2134 raft_consensus.cc:1905] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority I0109 07:21:21.927865 1996 leader_election.cc:347] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [CANDIDATE]: Term 7 pre-election: Vote granted by peer 7897578c54854ba0a5af32eb2aa04704 I0109 07:21:21.928087 2071 raft_consensus.cc:1691] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7897578c54854ba0a5af32eb2aa04704 for term 7 because replica has last-logged OpId of term: 6 index: 209, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 206. I0109 07:21:22.266629 2046 raft_consensus.cc:410] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0109 07:21:22.266721 2046 raft_consensus.cc:2158] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [term 6 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 0.166s I0109 07:21:22.266798 2046 raft_consensus.cc:434] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "710c236995db46699eb9055c19ac506e" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-4.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "1678276a36f849c28459de6b6788965a" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-5.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "591b1a1d383d4b659d7549653f4f136e" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-2.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "c5a3e756029444b2bf66ee0044f5bc35" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-1.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "7897578c54854ba0a5af32eb2aa04704" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-3.gce.cloudera.com" port: 7050 } } I0109 07:21:22.463908 2046 leader_election.cc:215] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [CANDIDATE]: Term 7 pre-election: Requesting pre-vote from peer 710c236995db46699eb9055c19ac506e I0109 07:21:22.464252 2046 leader_election.cc:215] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [CANDIDATE]: Term 7 pre-election: Requesting pre-vote from peer 591b1a1d383d4b659d7549653f4f136e W0109 07:21:22.464407 1995 leader_election.cc:271] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 710c236995db46699eb9055c19ac506e: Network error: Client connection negotiation failed: client connection to 172.31.114.38:7050: connect: Connection refused (error 111) I0109 07:21:22.464457 2046 leader_election.cc:215] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [CANDIDATE]: Term 7 pre-election: Requesting pre-vote from peer c5a3e756029444b2bf66ee0044f5bc35 I0109 07:21:22.464555 2046 leader_election.cc:215] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [CANDIDATE]: Term 7 pre-election: Requesting pre-vote from peer 7897578c54854ba0a5af32eb2aa04704 W0109 07:21:22.464850 1996 leader_election.cc:271] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 591b1a1d383d4b659d7549653f4f136e: Network error: Client connection negotiation failed: client connection to 172.31.113.2:7050: connect: Connection refused (error 111) W0109 07:21:22.464961 1993 leader_election.cc:271] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer c5a3e756029444b2bf66ee0044f5bc35: Network error: Client connection negotiation failed: client connection to 172.31.112.255:7050: connect: Connection refused (error 111) I0109 07:21:22.464979 1993 leader_election.cc:242] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. I0109 07:21:22.465140 2136 raft_consensus.cc:2158] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [term 6 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 0.590s I0109 07:21:22.465158 2136 raft_consensus.cc:1905] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority I0109 07:21:22.465307 1996 leader_election.cc:347] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [CANDIDATE]: Term 7 pre-election: Vote granted by peer 7897578c54854ba0a5af32eb2aa04704 I0109 07:21:23.229760 2046 raft_consensus.cc:410] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0109 07:21:23.229838 2046 raft_consensus.cc:2158] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [term 6 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.012s I0109 07:21:23.229909 2046 raft_consensus.cc:434] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "710c236995db46699eb9055c19ac506e" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-4.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "1678276a36f849c28459de6b6788965a" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-5.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "591b1a1d383d4b659d7549653f4f136e" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-2.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "c5a3e756029444b2bf66ee0044f5bc35" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-1.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "7897578c54854ba0a5af32eb2aa04704" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-3.gce.cloudera.com" port: 7050 } } I0109 07:21:23.426880 2071 raft_consensus.cc:1691] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7897578c54854ba0a5af32eb2aa04704 for term 7 because replica has last-logged OpId of term: 6 index: 209, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 206. I0109 07:21:23.427139 2046 leader_election.cc:215] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [CANDIDATE]: Term 7 pre-election: Requesting pre-vote from peer 710c236995db46699eb9055c19ac506e I0109 07:21:23.427299 2046 leader_election.cc:215] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [CANDIDATE]: Term 7 pre-election: Requesting pre-vote from peer 591b1a1d383d4b659d7549653f4f136e I0109 07:21:23.427409 2046 leader_election.cc:215] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [CANDIDATE]: Term 7 pre-election: Requesting pre-vote from peer c5a3e756029444b2bf66ee0044f5bc35 I0109 07:21:23.427537 2046 leader_election.cc:215] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [CANDIDATE]: Term 7 pre-election: Requesting pre-vote from peer 7897578c54854ba0a5af32eb2aa04704 W0109 07:21:23.428226 1995 leader_election.cc:271] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 710c236995db46699eb9055c19ac506e: Network error: Client connection negotiation failed: client connection to 172.31.114.38:7050: connect: Connection refused (error 111) W0109 07:21:23.428308 1996 leader_election.cc:271] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 591b1a1d383d4b659d7549653f4f136e: Network error: Client connection negotiation failed: client connection to 172.31.113.2:7050: connect: Connection refused (error 111) I0109 07:21:23.428349 1996 leader_election.cc:347] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [CANDIDATE]: Term 7 pre-election: Vote granted by peer 7897578c54854ba0a5af32eb2aa04704 W0109 07:21:23.428371 1993 leader_election.cc:271] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer c5a3e756029444b2bf66ee0044f5bc35: Network error: Client connection negotiation failed: client connection to 172.31.112.255:7050: connect: Connection refused (error 111) I0109 07:21:23.428380 1993 leader_election.cc:242] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. I0109 07:21:23.428534 2140 raft_consensus.cc:2158] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [term 6 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 0.887s I0109 07:21:23.428557 2140 raft_consensus.cc:1905] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority I0109 07:21:24.411738 2071 raft_consensus.cc:1691] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7897578c54854ba0a5af32eb2aa04704 for term 7 because replica has last-logged OpId of term: 6 index: 209, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 206. I0109 07:21:24.524452 2046 raft_consensus.cc:410] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0109 07:21:24.524529 2046 raft_consensus.cc:2158] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [term 6 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 0.689s I0109 07:21:24.524624 2046 raft_consensus.cc:434] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "710c236995db46699eb9055c19ac506e" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-4.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "1678276a36f849c28459de6b6788965a" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-5.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "591b1a1d383d4b659d7549653f4f136e" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-2.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "c5a3e756029444b2bf66ee0044f5bc35" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-1.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "7897578c54854ba0a5af32eb2aa04704" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-3.gce.cloudera.com" port: 7050 } } I0109 07:21:24.724175 2046 leader_election.cc:215] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [CANDIDATE]: Term 7 pre-election: Requesting pre-vote from peer 710c236995db46699eb9055c19ac506e I0109 07:21:24.724416 2046 leader_election.cc:215] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [CANDIDATE]: Term 7 pre-election: Requesting pre-vote from peer 591b1a1d383d4b659d7549653f4f136e I0109 07:21:24.724508 2046 leader_election.cc:215] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [CANDIDATE]: Term 7 pre-election: Requesting pre-vote from peer c5a3e756029444b2bf66ee0044f5bc35 I0109 07:21:24.724638 2046 leader_election.cc:215] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [CANDIDATE]: Term 7 pre-election: Requesting pre-vote from peer 7897578c54854ba0a5af32eb2aa04704 W0109 07:21:24.725301 1995 leader_election.cc:271] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 710c236995db46699eb9055c19ac506e: Network error: Client connection negotiation failed: client connection to 172.31.114.38:7050: connect: Connection refused (error 111) W0109 07:21:24.725347 1996 leader_election.cc:271] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 591b1a1d383d4b659d7549653f4f136e: Network error: Client connection negotiation failed: client connection to 172.31.113.2:7050: connect: Connection refused (error 111) W0109 07:21:24.725376 1993 leader_election.cc:271] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer c5a3e756029444b2bf66ee0044f5bc35: Network error: Client connection negotiation failed: client connection to 172.31.112.255:7050: connect: Connection refused (error 111) I0109 07:21:24.725385 1993 leader_election.cc:242] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. I0109 07:21:24.725672 1996 leader_election.cc:347] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [CANDIDATE]: Term 7 pre-election: Vote granted by peer 7897578c54854ba0a5af32eb2aa04704 I0109 07:21:24.725734 2145 raft_consensus.cc:2158] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [term 6 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 0.965s I0109 07:21:24.725749 2145 raft_consensus.cc:1905] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority I0109 07:21:25.810359 2071 raft_consensus.cc:2158] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [term 6 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.178s I0109 07:21:25.810421 2071 raft_consensus.cc:1744] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5a3e756029444b2bf66ee0044f5bc35 in term 6. I0109 07:21:25.944612 2071 raft_consensus.cc:2158] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [term 6 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 2.259s I0109 07:21:25.944658 2071 raft_consensus.cc:1744] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 591b1a1d383d4b659d7549653f4f136e in term 6. I0109 07:21:26.009830 2071 raft_consensus.cc:2202] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [term 6 FOLLOWER]: Advancing to term 7 I0109 07:21:26.009918 2071 raft_consensus.cc:2158] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [term 7 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 2.027s I0109 07:21:26.010294 2071 raft_consensus.cc:1744] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5a3e756029444b2bf66ee0044f5bc35 in term 7. I0109 07:21:26.142920 2071 raft_consensus.cc:1674] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 591b1a1d383d4b659d7549653f4f136e in current term 7: Already voted for candidate c5a3e756029444b2bf66ee0044f5bc35 in this term. I0109 07:21:26.210491 2071 raft_consensus.cc:915] T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [term 7 FOLLOWER]: Refusing update from remote peer c5a3e756029444b2bf66ee0044f5bc35: Log matching property violated. Preceding OpId in replica: term: 6 index: 209. Preceding OpId from leader: term: 7 index: 210. (index mismatch)