I0109 07:20:10.657807 29181 tablet_server_main.cc:56] Tablet server non-default flags:
--raft_heartbeat_interval_ms=50
--fs_data_dirs=/var/lib/kudu/tserver
--fs_wal_dir=/var/lib/kudu/tserver
--tserver_master_addrs=aserbin-kudu-jepsen-master.gce.cloudera.com:7051
--heap_profile_path=/tmp/kudu-tserver.29181
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--flagfile=/etc/kudu/conf/tserver.gflagfile
--log_dir=/var/log/kudu
--logtostderr=true
Tablet server version:
kudu 1.3.0-SNAPSHOT
revision 3a23b058af0f4de04f8827255de405fff7bf9ee1
build type RELEASE
built by jenkins at 09 Jan 2017 06:07:18 PST on aserbin-kudu-jepsen-control-node-debian8.gce.cloudera.com
build id 2017-01-09_06-07-06
I0109 07:20:10.668450 29181 mem_tracker.cc:140] MemTracker: hard memory limit is 0.464404 GB
I0109 07:20:10.668490 29181 mem_tracker.cc:142] MemTracker: soft memory limit is 0.278643 GB
I0109 07:20:10.746944 29181 tablet_server_main.cc:63] Initializing tablet server...
I0109 07:20:10.797047 29181 hybrid_clock.cc:177] HybridClock initialized. Resolution in nanos?: 1 Wait times tolerance adjustment: 1.0005 Current error: 900040
I0109 07:20:10.799156 29181 env_posix.cc:1286] Not raising process file limit of 65536; it is already as high as it can go
I0109 07:20:10.799188 29181 file_cache.cc:401] Constructed file cache lbm with capacity 32768
I0109 07:20:10.800549 29181 server_base.cc:179] Could not load existing FS layout: Not found: /var/lib/kudu/tserver/instance: No such file or directory (error 2)
I0109 07:20:10.800565 29181 server_base.cc:180] Creating new FS layout
I0109 07:20:10.807489 29181 fs_manager.cc:362] Generated new instance metadata in path /var/lib/kudu/tserver/instance:
uuid: "591b1a1d383d4b659d7549653f4f136e"
format_stamp: "Formatted at 2017-01-09 15:20:10 on aserbin-kudu-jepsen-ts-2"
I0109 07:20:10.814512 29181 fs_manager.cc:251] Opened local filesystem: /var/lib/kudu/tserver
uuid: "591b1a1d383d4b659d7549653f4f136e"
format_stamp: "Formatted at 2017-01-09 15:20:10 on aserbin-kudu-jepsen-ts-2"
I0109 07:20:10.832131 29181 tablet_server_main.cc:66] Starting tablet server...
I0109 07:20:10.863886 29181 rpc_server.cc:164] RPC server started. Bound to: 0.0.0.0:7050
I0109 07:20:10.863998 29181 webserver.cc:126] Starting webserver on 0.0.0.0:8050
I0109 07:20:10.864008 29181 webserver.cc:137] Document root disabled
I0109 07:20:10.864176 29181 webserver.cc:221] Webserver started. Bound to: http://0.0.0.0:8050/
I0109 07:20:10.864624 29181 tablet_server_main.cc:69] Tablet server successfully started.
W0109 07:20:10.917924 29316 heartbeater.cc:453] Failed to heartbeat to aserbin-kudu-jepsen-master.gce.cloudera.com:7051: Network error: Failed to ping master at aserbin-kudu-jepsen-master.gce.cloudera.com:7051: Client connection negotiation failed: client connection to 172.31.113.125:7051: connect: Connection refused (error 111)
W0109 07:20:10.967314 29316 heartbeater.cc:453] Failed to heartbeat to aserbin-kudu-jepsen-master.gce.cloudera.com:7051: Network error: Failed to ping master at aserbin-kudu-jepsen-master.gce.cloudera.com:7051: Client connection negotiation failed: client connection to 172.31.113.125:7051: connect: Connection refused (error 111)
W0109 07:20:11.016649 29316 heartbeater.cc:453] Failed to heartbeat to aserbin-kudu-jepsen-master.gce.cloudera.com:7051: Network error: Failed to ping master at aserbin-kudu-jepsen-master.gce.cloudera.com:7051: Client connection negotiation failed: client connection to 172.31.113.125:7051: connect: Connection refused (error 111)
W0109 07:20:11.016700 29316 heartbeater.cc:323] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I0109 07:20:12.067364 29316 heartbeater.cc:292] Connected to a master server at aserbin-kudu-jepsen-master.gce.cloudera.com:7051
I0109 07:20:12.067425 29316 heartbeater.cc:359] Registering TS with master...
I0109 07:20:12.067488 29316 heartbeater.cc:373] Master aserbin-kudu-jepsen-master.gce.cloudera.com:7051 requested a full tablet report, sending...
I0109 07:20:13.068087 29316 heartbeater.cc:365] Master aserbin-kudu-jepsen-master.gce.cloudera.com:7051 was elected leader, sending a full tablet report...
I0109 07:20:16.977461 29253 tablet_service.cc:630] Processing CreateTablet for tablet 9e99b710634a44f4a948132a5a180600 (table=rw-register-1483975178870 [id=0e0bbef280744ed1a8036e11eb0aaea0]), partition=RANGE (key) PARTITION UNBOUNDED
I0109 07:20:16.985558 29253 ts_tablet_manager.cc:863] Registered tablet 9e99b710634a44f4a948132a5a180600
I0109 07:20:16.985921 29334 ts_tablet_manager.cc:725] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e: Bootstrapping tablet
I0109 07:20:16.986053 29334 tablet_bootstrap.cc:381] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e: Bootstrap starting.
I0109 07:20:16.986908 29334 tablet_bootstrap.cc:540] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e: Time spent opening tablet: real 0.000s	user 0.000s	sys 0.000s
I0109 07:20:16.987013 29334 tablet_bootstrap.cc:483] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e: No blocks or log segments found. Creating new log.
I0109 07:20:16.987138 29334 log.cc:351] Log is configured to *not* fsync() on all Append() calls
I0109 07:20:16.987522 29334 tablet_bootstrap.cc:381] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e: No bootstrap required, opened a new log
I0109 07:20:16.987550 29334 ts_tablet_manager.cc:730] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e: Time spent bootstrapping tablet: real 0.002s	user 0.000s	sys 0.000s
I0109 07:20:16.988778 29334 raft_consensus.cc:287] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "710c236995db46699eb9055c19ac506e" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-4.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "1678276a36f849c28459de6b6788965a" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-5.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "591b1a1d383d4b659d7549653f4f136e" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-2.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "c5a3e756029444b2bf66ee0044f5bc35" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-1.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "7897578c54854ba0a5af32eb2aa04704" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-3.gce.cloudera.com" port: 7050 } }
I0109 07:20:16.988807 29334 raft_consensus.cc:318] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0109 07:20:16.988821 29334 raft_consensus.cc:547] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 591b1a1d383d4b659d7549653f4f136e, State: 1, Role: FOLLOWER
I0109 07:20:16.988837 29334 consensus_queue.cc:175] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER
I0109 07:20:16.988978 29334 ts_tablet_manager.cc:757] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e: Time spent starting tablet: real 0.001s	user 0.000s	sys 0.000s
I0109 07:20:17.057847 29337 raft_consensus.cc:410] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0109 07:20:17.057937 29337 raft_consensus.cc:2158] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [term 0 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 0.197s
I0109 07:20:17.058007 29337 raft_consensus.cc:434] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "710c236995db46699eb9055c19ac506e" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-4.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "1678276a36f849c28459de6b6788965a" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-5.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "591b1a1d383d4b659d7549653f4f136e" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-2.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "c5a3e756029444b2bf66ee0044f5bc35" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-1.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "7897578c54854ba0a5af32eb2aa04704" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-3.gce.cloudera.com" port: 7050 } }
I0109 07:20:17.255017 29273 raft_consensus.cc:2158] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [term 0 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 0.170s
I0109 07:20:17.255079 29273 raft_consensus.cc:1744] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 710c236995db46699eb9055c19ac506e in term 0.
I0109 07:20:17.255209 29272 raft_consensus.cc:1719] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [term 0 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1678276a36f849c28459de6b6788965a for term 1 because replica is already servicing an update from a current leader or another vote.
I0109 07:20:17.255460 29337 leader_election.cc:215] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 710c236995db46699eb9055c19ac506e
I0109 07:20:17.255698 29337 leader_election.cc:215] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 1678276a36f849c28459de6b6788965a
I0109 07:20:17.255877 29337 leader_election.cc:215] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer c5a3e756029444b2bf66ee0044f5bc35
I0109 07:20:17.256011 29337 leader_election.cc:215] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 7897578c54854ba0a5af32eb2aa04704
I0109 07:20:17.257164 29202 leader_election.cc:347] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [CANDIDATE]: Term 1 pre-election: Vote granted by peer 710c236995db46699eb9055c19ac506e
I0109 07:20:17.257596 29200 leader_election.cc:347] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [CANDIDATE]: Term 1 pre-election: Vote granted by peer 1678276a36f849c28459de6b6788965a
I0109 07:20:17.257616 29200 leader_election.cc:242] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won.
W0109 07:20:17.257737 29273 rpcz_store.cc:234] Call kudu.consensus.ConsensusService.RequestConsensusVote from 172.31.114.38:34808 (request call id 1) took 197ms (client timeout 203).
W0109 07:20:17.257798 29273 rpcz_store.cc:238] Trace:
0109 07:20:17.060171 (+     0us) service_pool.cc:143] Inserting onto call queue
0109 07:20:17.060181 (+    10us) service_pool.cc:202] Handling call
0109 07:20:17.257723 (+197542us) spinlock_profiling.cc:255] Waited 195 ms on lock 0x3f2f750. stack: 0000000001a20b21 00000000009d8b77 00000000007e48af 0000000000a8971f 0000000000a8a342 0000000001a2dc19 00007ff86454d0a3 00007ff862ddd87c ffffffffffffffff
0109 07:20:17.257728 (+     5us) inbound_call.cc:130] Queueing success response
Metrics: {"spinlock_wait_cycles":486845184}
I0109 07:20:17.258206 29342 raft_consensus.cc:2158] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [term 0 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 0.165s
I0109 07:20:17.258239 29342 raft_consensus.cc:1968] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [term 0 FOLLOWER]: Leader pre-election won for term 1
I0109 07:20:17.258247 29342 raft_consensus.cc:410] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I0109 07:20:17.258257 29342 raft_consensus.cc:2158] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [term 0 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 0.196s
I0109 07:20:17.258263 29342 raft_consensus.cc:2202] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [term 0 FOLLOWER]: Advancing to term 1
I0109 07:20:17.258641 29342 raft_consensus.cc:434] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "710c236995db46699eb9055c19ac506e" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-4.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "1678276a36f849c28459de6b6788965a" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-5.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "591b1a1d383d4b659d7549653f4f136e" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-2.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "c5a3e756029444b2bf66ee0044f5bc35" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-1.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "7897578c54854ba0a5af32eb2aa04704" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-3.gce.cloudera.com" port: 7050 } }
I0109 07:20:17.331892 29199 leader_election.cc:361] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [CANDIDATE]: Term 1 pre-election: Vote denied by peer c5a3e756029444b2bf66ee0044f5bc35. Message: Service unavailable: T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35 [term 0 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 591b1a1d383d4b659d7549653f4f136e for term 1 because replica is already servicing an update from a current leader or another vote.
I0109 07:20:17.420505 29204 leader_election.cc:347] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [CANDIDATE]: Term 1 pre-election: Vote granted by peer 7897578c54854ba0a5af32eb2aa04704
I0109 07:20:17.457098 29272 raft_consensus.cc:1719] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7897578c54854ba0a5af32eb2aa04704 for term 1 because replica is already servicing an update from a current leader or another vote.
I0109 07:20:17.457291 29273 raft_consensus.cc:1674] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c5a3e756029444b2bf66ee0044f5bc35 in current term 1: Already voted for candidate 591b1a1d383d4b659d7549653f4f136e in this term.
I0109 07:20:17.457356 29271 raft_consensus.cc:1719] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 710c236995db46699eb9055c19ac506e for term 1 because replica is already servicing an update from a current leader or another vote.
I0109 07:20:17.457510 29342 leader_election.cc:215] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [CANDIDATE]: Term 1 election: Requesting vote from peer 710c236995db46699eb9055c19ac506e
I0109 07:20:17.457595 29342 leader_election.cc:215] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [CANDIDATE]: Term 1 election: Requesting vote from peer 1678276a36f849c28459de6b6788965a
I0109 07:20:17.457660 29342 leader_election.cc:215] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [CANDIDATE]: Term 1 election: Requesting vote from peer c5a3e756029444b2bf66ee0044f5bc35
I0109 07:20:17.457726 29342 leader_election.cc:215] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [CANDIDATE]: Term 1 election: Requesting vote from peer 7897578c54854ba0a5af32eb2aa04704
I0109 07:20:17.457947 29200 leader_election.cc:361] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [CANDIDATE]: Term 1 election: Vote denied by peer 1678276a36f849c28459de6b6788965a. Message: Invalid argument: T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 591b1a1d383d4b659d7549653f4f136e in current term 1: Already voted for candidate 710c236995db46699eb9055c19ac506e in this term.
I0109 07:20:17.458065 29199 leader_election.cc:361] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [CANDIDATE]: Term 1 election: Vote denied by peer c5a3e756029444b2bf66ee0044f5bc35. Message: Invalid argument: T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 591b1a1d383d4b659d7549653f4f136e in current term 1: Already voted for candidate 710c236995db46699eb9055c19ac506e in this term.
I0109 07:20:17.607115 29337 raft_consensus.cc:410] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0109 07:20:17.607198 29337 raft_consensus.cc:2158] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 0.194s
I0109 07:20:17.607267 29337 raft_consensus.cc:434] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "710c236995db46699eb9055c19ac506e" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-4.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "1678276a36f849c28459de6b6788965a" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-5.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "591b1a1d383d4b659d7549653f4f136e" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-2.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "c5a3e756029444b2bf66ee0044f5bc35" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-1.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "7897578c54854ba0a5af32eb2aa04704" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-3.gce.cloudera.com" port: 7050 } }
I0109 07:20:17.622148 29204 leader_election.cc:361] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [CANDIDATE]: Term 1 election: Vote denied by peer 7897578c54854ba0a5af32eb2aa04704. Message: Service unavailable: T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 591b1a1d383d4b659d7549653f4f136e for term 1 because replica is already servicing an update from a current leader or another vote.
I0109 07:20:17.622220 29204 leader_election.cc:242] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost.
W0109 07:20:17.654208 29202 leader_election.cc:271] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 710c236995db46699eb9055c19ac506e: Timed out: RequestConsensusVote RPC to 172.31.114.38:7050 timed out after 0.196s (SENT)
I0109 07:20:17.804137 29337 leader_election.cc:215] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 710c236995db46699eb9055c19ac506e
I0109 07:20:17.804224 29337 leader_election.cc:215] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 1678276a36f849c28459de6b6788965a
I0109 07:20:17.804365 29271 raft_consensus.cc:1674] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 7897578c54854ba0a5af32eb2aa04704 in current term 1: Already voted for candidate 591b1a1d383d4b659d7549653f4f136e in this term.
W0109 07:20:17.804483 29271 rpcz_store.cc:234] Call kudu.consensus.ConsensusService.RequestConsensusVote from 172.31.114.37:60784 (request call id 2) took 182ms (client timeout 153).
W0109 07:20:17.804544 29271 rpcz_store.cc:238] Trace:
0109 07:20:17.622397 (+     0us) service_pool.cc:143] Inserting onto call queue
0109 07:20:17.622410 (+    13us) service_pool.cc:202] Handling call
0109 07:20:17.804455 (+182045us) spinlock_profiling.cc:255] Waited 182 ms on lock 0x3f2f750. stack: 0000000001a20b21 00000000009d8b77 00000000007e48af 0000000000a8971f 0000000000a8a342 0000000001a2dc19 00007ff86454d0a3 00007ff862ddd87c ffffffffffffffff
0109 07:20:17.804468 (+    13us) inbound_call.cc:130] Queueing success response
Metrics: {"spinlock_wait_cycles":454801792}
I0109 07:20:17.804667 29273 raft_consensus.cc:915] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [term 1 FOLLOWER]: Refusing update from remote peer 710c236995db46699eb9055c19ac506e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch)
I0109 07:20:17.804846 29200 leader_election.cc:361] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [CANDIDATE]: Term 2 pre-election: Vote denied by peer 1678276a36f849c28459de6b6788965a. Message: Invalid argument: T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 591b1a1d383d4b659d7549653f4f136e for term 2 because replica is either leader or believes a valid leader to be alive.
I0109 07:20:17.804898 29337 leader_election.cc:215] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer c5a3e756029444b2bf66ee0044f5bc35
I0109 07:20:17.804947 29337 leader_election.cc:215] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 7897578c54854ba0a5af32eb2aa04704
I0109 07:20:17.805021 29343 raft_consensus.cc:2158] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 0.222s
I0109 07:20:17.805042 29343 raft_consensus.cc:1905] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I0109 07:20:17.805335 29204 leader_election.cc:361] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [CANDIDATE]: Term 2 pre-election: Vote denied by peer 7897578c54854ba0a5af32eb2aa04704. Message: Invalid argument: T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 591b1a1d383d4b659d7549653f4f136e for term 2 because replica is either leader or believes a valid leader to be alive.
I0109 07:20:17.805424 29202 leader_election.cc:361] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [CANDIDATE]: Term 2 pre-election: Vote denied by peer 710c236995db46699eb9055c19ac506e. Message: Invalid argument: T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate 591b1a1d383d4b659d7549653f4f136e for term 2 because replica is either leader or believes a valid leader to be alive.
I0109 07:20:17.805438 29202 leader_election.cc:242] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost.
I0109 07:20:17.805465 29343 raft_consensus.cc:2158] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 0.164s
I0109 07:20:17.805481 29343 raft_consensus.cc:1905] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I0109 07:20:17.805613 29199 leader_election.cc:361] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [CANDIDATE]: Term 2 pre-election: Vote denied by peer c5a3e756029444b2bf66ee0044f5bc35. Message: Invalid argument: T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 591b1a1d383d4b659d7549653f4f136e for term 2 because replica is either leader or believes a valid leader to be alive.
I0109 07:20:30.609977 29360 tablet_server_main.cc:56] Tablet server non-default flags:
--raft_heartbeat_interval_ms=50
--fs_data_dirs=/var/lib/kudu/tserver
--fs_wal_dir=/var/lib/kudu/tserver
--tserver_master_addrs=aserbin-kudu-jepsen-master.gce.cloudera.com:7051
--heap_profile_path=/tmp/kudu-tserver.29360
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--flagfile=/etc/kudu/conf/tserver.gflagfile
--log_dir=/var/log/kudu
--logtostderr=true
Tablet server version:
kudu 1.3.0-SNAPSHOT
revision 3a23b058af0f4de04f8827255de405fff7bf9ee1
build type RELEASE
built by jenkins at 09 Jan 2017 06:07:18 PST on aserbin-kudu-jepsen-control-node-debian8.gce.cloudera.com
build id 2017-01-09_06-07-06
I0109 07:20:30.610200 29360 mem_tracker.cc:140] MemTracker: hard memory limit is 0.464404 GB
I0109 07:20:30.610219 29360 mem_tracker.cc:142] MemTracker: soft memory limit is 0.278643 GB
I0109 07:20:30.611889 29360 tablet_server_main.cc:63] Initializing tablet server...
I0109 07:20:30.662029 29360 hybrid_clock.cc:177] HybridClock initialized. Resolution in nanos?: 1 Wait times tolerance adjustment: 1.0005 Current error: 910040
I0109 07:20:30.662863 29360 env_posix.cc:1286] Not raising process file limit of 65536; it is already as high as it can go
I0109 07:20:30.662900 29360 file_cache.cc:401] Constructed file cache lbm with capacity 32768
I0109 07:20:30.664053 29360 fs_manager.cc:251] Opened local filesystem: /var/lib/kudu/tserver
uuid: "591b1a1d383d4b659d7549653f4f136e"
format_stamp: "Formatted at 2017-01-09 15:20:10 on aserbin-kudu-jepsen-ts-2"
I0109 07:20:30.667335 29360 ts_tablet_manager.cc:701] Loading metadata for tablet 9e99b710634a44f4a948132a5a180600
I0109 07:20:30.667626 29360 ts_tablet_manager.cc:863] Registered tablet 9e99b710634a44f4a948132a5a180600
I0109 07:20:30.667817 29374 ts_tablet_manager.cc:725] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e: Bootstrapping tablet
I0109 07:20:30.668148 29374 tablet_bootstrap.cc:381] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e: Bootstrap starting.
I0109 07:20:30.668422 29374 tablet_bootstrap.cc:540] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e: Time spent opening tablet: real 0.000s	user 0.000s	sys 0.000s
I0109 07:20:30.668510 29374 tablet_bootstrap.cc:596] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e: Will attempt to recover log segment /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600/wal-000000001 to /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery/wal-000000001
I0109 07:20:30.668524 29374 tablet_bootstrap.cc:604] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e: Moving log directory /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600 to recovery directory /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery in preparation for log replay
I0109 07:20:30.668761 29374 log_util.cc:315] Log segment /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery/wal-000000001 has no footer. This segment was likely being written when the server previously shut down.
I0109 07:20:30.668772 29374 log_reader.cc:160] Log segment /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery/wal-000000001 was likely left in-progress after a previous crash. Will try to rebuild footer by scanning data.
I0109 07:20:30.670163 29360 tablet_server_main.cc:66] Starting tablet server...
I0109 07:20:30.671711 29374 log_util.cc:569] Scanning /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery/wal-000000001 for valid entry headers following offset 10526...
I0109 07:20:30.751657 29374 log_util.cc:606] Found no log entry headers
I0109 07:20:30.751718 29374 log_util.cc:218] Ignoring log segment corruption in /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery/wal-000000001 because there are no log entries following the corrupted one. The server probably crashed in the middle of writing an entry to the write-ahead log or downloaded an active log via tablet copy. Error detail: Corruption: CRC mismatch in log entry header: Log file corruption detected. Failed trying to read batch #0 at offset 10514 for log segment /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery/wal-000000001: Prior entries: [off=10249 REPLICATE (1.45)] [off=10283 COMMIT (1.45)] [off=10480 REPLICATE (1.46)] [off=10514 COMMIT (1.46)]
I0109 07:20:30.751727 29374 log_util.cc:368] Successfully rebuilt footer for segment: /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery/wal-000000001 (valid entries through byte offset 10514)
I0109 07:20:30.751791 29374 tablet.cc:983] T 9e99b710634a44f4a948132a5a180600 Rewinding schema during bootstrap to Schema [
	0:key[string NOT NULL],
	1:value[int32 NOT NULL]
]
I0109 07:20:30.752060 29374 log.cc:351] Log is configured to *not* fsync() on all Append() calls
I0109 07:20:30.754611 29374 tablet_bootstrap.cc:381] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e: Bootstrap replayed 1/1 log segments. Stats: ops{read=46 overwritten=0 applied=46 ignored=0} inserts{seen=45 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I0109 07:20:30.754637 29374 tablet_bootstrap.cc:1017] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e: ReplayState: Previous OpId: 1.46, Committed OpId: 1.46, Pending Replicates: 0, Pending Commits: 0
I0109 07:20:30.754947 29374 tablet_bootstrap.cc:633] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e: Preparing to delete log recovery files and directory /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery
I0109 07:20:30.754958 29374 tablet_bootstrap.cc:636] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e: Renaming log recovery dir from /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery to /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery-1483975230754958
I0109 07:20:30.754977 29374 tablet_bootstrap.cc:646] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e: Deleting all files from renamed log recovery directory /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery-1483975230754958
I0109 07:20:30.755059 29374 tablet_bootstrap.cc:649] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e: Completed deletion of old log recovery files and directory /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery-1483975230754958
I0109 07:20:30.755069 29374 tablet_bootstrap.cc:381] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e: Bootstrap complete.
I0109 07:20:30.760321 29374 ts_tablet_manager.cc:730] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e: Time spent bootstrapping tablet: real 0.092s	user 0.008s	sys 0.036s
I0109 07:20:30.761131 29374 raft_consensus.cc:287] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "710c236995db46699eb9055c19ac506e" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-4.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "1678276a36f849c28459de6b6788965a" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-5.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "591b1a1d383d4b659d7549653f4f136e" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-2.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "c5a3e756029444b2bf66ee0044f5bc35" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-1.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "7897578c54854ba0a5af32eb2aa04704" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-3.gce.cloudera.com" port: 7050 } }
I0109 07:20:30.761175 29374 raft_consensus.cc:547] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 591b1a1d383d4b659d7549653f4f136e, State: 1, Role: FOLLOWER
I0109 07:20:30.761193 29374 consensus_queue.cc:175] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 46, Last appended: 1.46, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER
I0109 07:20:30.761365 29374 ts_tablet_manager.cc:757] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e: Time spent starting tablet: real 0.001s	user 0.000s	sys 0.000s
I0109 07:20:30.768486 29360 rpc_server.cc:164] RPC server started. Bound to: 0.0.0.0:7050
I0109 07:20:30.768568 29360 webserver.cc:126] Starting webserver on 0.0.0.0:8050
I0109 07:20:30.768576 29360 webserver.cc:137] Document root disabled
I0109 07:20:30.768743 29360 webserver.cc:221] Webserver started. Bound to: http://0.0.0.0:8050/
I0109 07:20:30.769131 29360 tablet_server_main.cc:69] Tablet server successfully started.
I0109 07:20:30.820521 29490 heartbeater.cc:292] Connected to a master server at aserbin-kudu-jepsen-master.gce.cloudera.com:7051
I0109 07:20:30.820567 29490 heartbeater.cc:359] Registering TS with master...
I0109 07:20:30.820641 29490 heartbeater.cc:373] Master aserbin-kudu-jepsen-master.gce.cloudera.com:7051 requested a full tablet report, sending...
I0109 07:20:30.912459 29414 raft_consensus.cc:410] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0109 07:20:30.912556 29414 raft_consensus.cc:2158] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 0.152s
I0109 07:20:30.912638 29414 raft_consensus.cc:434] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "710c236995db46699eb9055c19ac506e" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-4.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "1678276a36f849c28459de6b6788965a" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-5.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "591b1a1d383d4b659d7549653f4f136e" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-2.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "c5a3e756029444b2bf66ee0044f5bc35" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-1.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "7897578c54854ba0a5af32eb2aa04704" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-3.gce.cloudera.com" port: 7050 } }
I0109 07:20:31.109364 29414 leader_election.cc:215] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 710c236995db46699eb9055c19ac506e
I0109 07:20:31.109594 29414 leader_election.cc:215] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 1678276a36f849c28459de6b6788965a
I0109 07:20:31.109726 29414 leader_election.cc:215] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer c5a3e756029444b2bf66ee0044f5bc35
I0109 07:20:31.109887 29414 leader_election.cc:215] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 7897578c54854ba0a5af32eb2aa04704
W0109 07:20:31.111011 29370 leader_election.cc:271] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1678276a36f849c28459de6b6788965a: Network error: Client connection negotiation failed: client connection to 172.31.114.39:7050: connect: Connection refused (error 111)
I0109 07:20:31.112845 29369 leader_election.cc:361] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [CANDIDATE]: Term 2 pre-election: Vote denied by peer c5a3e756029444b2bf66ee0044f5bc35. Message: Invalid argument: T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 591b1a1d383d4b659d7549653f4f136e for term 2 because replica has last-logged OpId of term: 1 index: 49, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 46.
I0109 07:20:31.206692 29372 leader_election.cc:361] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [CANDIDATE]: Term 2 pre-election: Vote denied by peer 7897578c54854ba0a5af32eb2aa04704. Message: Invalid argument: T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 591b1a1d383d4b659d7549653f4f136e for term 2 because replica has last-logged OpId of term: 1 index: 48, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 46.
I0109 07:20:31.206771 29372 leader_election.cc:242] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost.
I0109 07:20:31.206954 29496 raft_consensus.cc:2158] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 0.219s
I0109 07:20:31.206969 29496 raft_consensus.cc:1905] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I0109 07:20:31.208149 29447 raft_consensus.cc:2158] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 0.229s
I0109 07:20:31.208171 29447 raft_consensus.cc:1744] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7897578c54854ba0a5af32eb2aa04704 in term 1.
W0109 07:20:31.262600 29371 leader_election.cc:271] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 710c236995db46699eb9055c19ac506e: Timed out: RequestConsensusVote RPC to 172.31.114.38:7050 timed out after 0.152s (SENT)
I0109 07:20:31.277065 29447 raft_consensus.cc:2158] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 0.226s
I0109 07:20:31.277107 29447 raft_consensus.cc:1744] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 710c236995db46699eb9055c19ac506e in term 1.
I0109 07:20:31.476872 29447 raft_consensus.cc:2202] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [term 1 FOLLOWER]: Advancing to term 2
I0109 07:20:31.476966 29447 raft_consensus.cc:2158] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [term 2 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 0.191s
I0109 07:20:31.477396 29447 raft_consensus.cc:1744] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 710c236995db46699eb9055c19ac506e in term 2.
I0109 07:20:31.821583 29490 heartbeater.cc:365] Master aserbin-kudu-jepsen-master.gce.cloudera.com:7051 was elected leader, sending a full tablet report...
I0109 07:20:31.836423 29414 raft_consensus.cc:410] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0109 07:20:31.836500 29414 raft_consensus.cc:2158] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [term 2 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 0.235s
I0109 07:20:31.836565 29414 raft_consensus.cc:434] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "710c236995db46699eb9055c19ac506e" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-4.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "1678276a36f849c28459de6b6788965a" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-5.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "591b1a1d383d4b659d7549653f4f136e" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-2.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "c5a3e756029444b2bf66ee0044f5bc35" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-1.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "7897578c54854ba0a5af32eb2aa04704" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-3.gce.cloudera.com" port: 7050 } }
I0109 07:20:32.034077 29414 leader_election.cc:215] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [CANDIDATE]: Term 3 pre-election: Requesting pre-vote from peer 710c236995db46699eb9055c19ac506e
I0109 07:20:32.034231 29414 leader_election.cc:215] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [CANDIDATE]: Term 3 pre-election: Requesting pre-vote from peer 1678276a36f849c28459de6b6788965a
I0109 07:20:32.034405 29414 leader_election.cc:215] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [CANDIDATE]: Term 3 pre-election: Requesting pre-vote from peer c5a3e756029444b2bf66ee0044f5bc35
I0109 07:20:32.034492 29414 leader_election.cc:215] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [CANDIDATE]: Term 3 pre-election: Requesting pre-vote from peer 7897578c54854ba0a5af32eb2aa04704
I0109 07:20:32.041476 29370 leader_election.cc:361] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [CANDIDATE]: Term 3 pre-election: Vote denied by peer 1678276a36f849c28459de6b6788965a. Message: Invalid argument: T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 591b1a1d383d4b659d7549653f4f136e for term 3 because replica has last-logged OpId of term: 1 index: 49, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 46.
I0109 07:20:32.043061 29447 raft_consensus.cc:2158] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [term 2 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 0.190s
I0109 07:20:32.043082 29447 raft_consensus.cc:1744] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1678276a36f849c28459de6b6788965a in term 2.
I0109 07:20:32.071884 29369 leader_election.cc:361] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [CANDIDATE]: Term 3 pre-election: Vote denied by peer c5a3e756029444b2bf66ee0044f5bc35. Message: Invalid argument: T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 591b1a1d383d4b659d7549653f4f136e for term 3 because replica has last-logged OpId of term: 1 index: 49, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 46.
I0109 07:20:32.074349 29447 raft_consensus.cc:2158] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [term 2 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 0.274s
I0109 07:20:32.074375 29447 raft_consensus.cc:1744] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5a3e756029444b2bf66ee0044f5bc35 in term 2.
I0109 07:20:32.172904 29447 raft_consensus.cc:2158] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [term 2 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 0.234s
I0109 07:20:32.172960 29447 raft_consensus.cc:1744] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7897578c54854ba0a5af32eb2aa04704 in term 2.
I0109 07:20:32.173125 29372 leader_election.cc:361] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [CANDIDATE]: Term 3 pre-election: Vote denied by peer 7897578c54854ba0a5af32eb2aa04704. Message: Invalid argument: T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 591b1a1d383d4b659d7549653f4f136e for term 3 because replica has last-logged OpId of term: 1 index: 48, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 46.
I0109 07:20:32.173151 29372 leader_election.cc:242] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost.
I0109 07:20:32.173317 29506 raft_consensus.cc:2158] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [term 2 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 0.302s
I0109 07:20:32.173332 29506 raft_consensus.cc:1905] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W0109 07:20:32.270043 29371 leader_election.cc:271] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 710c236995db46699eb9055c19ac506e: Timed out: RequestConsensusVote RPC to 172.31.114.38:7050 timed out after 0.235s (SENT)
I0109 07:20:32.627041 29414 raft_consensus.cc:410] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0109 07:20:32.627121 29414 raft_consensus.cc:2158] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [term 2 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 0.317s
I0109 07:20:32.627192 29414 raft_consensus.cc:434] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "710c236995db46699eb9055c19ac506e" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-4.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "1678276a36f849c28459de6b6788965a" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-5.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "591b1a1d383d4b659d7549653f4f136e" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-2.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "c5a3e756029444b2bf66ee0044f5bc35" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-1.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "7897578c54854ba0a5af32eb2aa04704" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-3.gce.cloudera.com" port: 7050 } }
I0109 07:20:32.824048 29447 raft_consensus.cc:2202] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [term 2 FOLLOWER]: Advancing to term 3
I0109 07:20:32.824120 29447 raft_consensus.cc:2158] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [term 3 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 0.205s
I0109 07:20:32.824462 29447 raft_consensus.cc:1744] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5a3e756029444b2bf66ee0044f5bc35 in term 3.
W0109 07:20:32.824615 29447 rpcz_store.cc:234] Call kudu.consensus.ConsensusService.RequestConsensusVote from 172.31.112.255:41899 (request call id 2) took 185ms (client timeout 164).
W0109 07:20:32.824688 29447 rpcz_store.cc:238] Trace:
0109 07:20:32.639029 (+     0us) service_pool.cc:143] Inserting onto call queue
0109 07:20:32.639052 (+    23us) service_pool.cc:202] Handling call
0109 07:20:32.824592 (+185540us) spinlock_profiling.cc:255] Waited 185 ms on lock 0x3dbcf70. stack: 0000000001a20b21 00000000009d8b77 00000000007e48af 0000000000a8971f 0000000000a8a342 0000000001a2dc19 00007f2122dae0a3 00007f212163e87c ffffffffffffffff
0109 07:20:32.824600 (+     8us) inbound_call.cc:130] Queueing success response
Metrics: {"spinlock_wait_cycles":462376704}
I0109 07:20:32.824728 29446 raft_consensus.cc:1719] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7897578c54854ba0a5af32eb2aa04704 for term 3 because replica is already servicing an update from a current leader or another vote.
I0109 07:20:32.824867 29414 leader_election.cc:215] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [CANDIDATE]: Term 3 pre-election: Requesting pre-vote from peer 710c236995db46699eb9055c19ac506e
I0109 07:20:32.824924 29414 leader_election.cc:215] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [CANDIDATE]: Term 3 pre-election: Requesting pre-vote from peer 1678276a36f849c28459de6b6788965a
I0109 07:20:32.824968 29414 leader_election.cc:215] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [CANDIDATE]: Term 3 pre-election: Requesting pre-vote from peer c5a3e756029444b2bf66ee0044f5bc35
I0109 07:20:32.825021 29414 leader_election.cc:215] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [CANDIDATE]: Term 3 pre-election: Requesting pre-vote from peer 7897578c54854ba0a5af32eb2aa04704
I0109 07:20:32.825469 29370 leader_election.cc:361] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [CANDIDATE]: Term 3 pre-election: Vote denied by peer 1678276a36f849c28459de6b6788965a. Message: Invalid argument: T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 591b1a1d383d4b659d7549653f4f136e in current term 3: Already voted for candidate c5a3e756029444b2bf66ee0044f5bc35 in this term.
I0109 07:20:32.825506 29372 leader_election.cc:361] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [CANDIDATE]: Term 3 pre-election: Vote denied by peer 7897578c54854ba0a5af32eb2aa04704. Message: Invalid argument: T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 591b1a1d383d4b659d7549653f4f136e in current term 3: Already voted for candidate c5a3e756029444b2bf66ee0044f5bc35 in this term.
I0109 07:20:32.825805 29446 raft_consensus.cc:1674] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1678276a36f849c28459de6b6788965a in current term 3: Already voted for candidate c5a3e756029444b2bf66ee0044f5bc35 in this term.
I0109 07:20:32.836242 29446 raft_consensus.cc:2158] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [term 3 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 0.271s
I0109 07:20:32.836305 29446 raft_consensus.cc:1744] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5a3e756029444b2bf66ee0044f5bc35 in term 3.
I0109 07:20:32.836369 29369 leader_election.cc:361] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [CANDIDATE]: Term 3 pre-election: Vote denied by peer c5a3e756029444b2bf66ee0044f5bc35. Message: Service unavailable: T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 591b1a1d383d4b659d7549653f4f136e for term 3 because replica is already servicing an update from a current leader or another vote.
I0109 07:20:32.836382 29369 leader_election.cc:242] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost.
I0109 07:20:32.836661 29507 raft_consensus.cc:2158] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [term 3 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 0.457s
I0109 07:20:32.836678 29507 raft_consensus.cc:1905] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W0109 07:20:33.142796 29371 leader_election.cc:271] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 710c236995db46699eb9055c19ac506e: Timed out: RequestConsensusVote RPC to 172.31.114.38:7050 timed out after 0.317s (SENT)
I0109 07:20:33.469704 29414 raft_consensus.cc:410] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0109 07:20:33.469785 29414 raft_consensus.cc:2158] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [term 3 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 0.444s
I0109 07:20:33.469852 29414 raft_consensus.cc:434] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "710c236995db46699eb9055c19ac506e" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-4.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "1678276a36f849c28459de6b6788965a" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-5.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "591b1a1d383d4b659d7549653f4f136e" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-2.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "c5a3e756029444b2bf66ee0044f5bc35" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-1.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "7897578c54854ba0a5af32eb2aa04704" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-3.gce.cloudera.com" port: 7050 } }
I0109 07:20:33.665841 29446 raft_consensus.cc:2158] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [term 3 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 0.705s
I0109 07:20:33.665889 29446 raft_consensus.cc:1744] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1678276a36f849c28459de6b6788965a in term 3.
I0109 07:20:33.665983 29447 raft_consensus.cc:1719] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7897578c54854ba0a5af32eb2aa04704 for term 4 because replica is already servicing an update from a current leader or another vote.
I0109 07:20:33.666147 29414 leader_election.cc:215] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [CANDIDATE]: Term 4 pre-election: Requesting pre-vote from peer 710c236995db46699eb9055c19ac506e
I0109 07:20:33.666227 29414 leader_election.cc:215] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [CANDIDATE]: Term 4 pre-election: Requesting pre-vote from peer 1678276a36f849c28459de6b6788965a
I0109 07:20:33.666272 29414 leader_election.cc:215] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [CANDIDATE]: Term 4 pre-election: Requesting pre-vote from peer c5a3e756029444b2bf66ee0044f5bc35
I0109 07:20:33.666345 29414 leader_election.cc:215] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [CANDIDATE]: Term 4 pre-election: Requesting pre-vote from peer 7897578c54854ba0a5af32eb2aa04704
I0109 07:20:33.666785 29372 leader_election.cc:361] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [CANDIDATE]: Term 4 pre-election: Vote denied by peer 7897578c54854ba0a5af32eb2aa04704. Message: Invalid argument: T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 591b1a1d383d4b659d7549653f4f136e for term 4 because replica has last-logged OpId of term: 1 index: 48, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 46.
I0109 07:20:33.666860 29370 leader_election.cc:361] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [CANDIDATE]: Term 4 pre-election: Vote denied by peer 1678276a36f849c28459de6b6788965a. Message: Invalid argument: T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 591b1a1d383d4b659d7549653f4f136e for term 4 because replica has last-logged OpId of term: 1 index: 49, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 46.
I0109 07:20:33.864814 29447 raft_consensus.cc:2202] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [term 3 FOLLOWER]: Advancing to term 4
I0109 07:20:33.864912 29447 raft_consensus.cc:2158] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [term 4 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 0.386s
I0109 07:20:33.865278 29447 raft_consensus.cc:1744] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1678276a36f849c28459de6b6788965a in term 4.
I0109 07:20:34.106195 29447 raft_consensus.cc:915] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [term 4 FOLLOWER]: Refusing update from remote peer 1678276a36f849c28459de6b6788965a: Log matching property violated. Preceding OpId in replica: term: 1 index: 46. Preceding OpId from leader: term: 4 index: 50. (index mismatch)
W0109 07:20:34.110288 29371 leader_election.cc:271] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 710c236995db46699eb9055c19ac506e: Timed out: RequestConsensusVote RPC to 172.31.114.38:7050 timed out after 0.444s (SENT)
I0109 07:20:34.110328 29371 leader_election.cc:242] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost.
I0109 07:20:34.110378 29509 raft_consensus.cc:2158] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [term 4 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 0.215s
I0109 07:20:34.110390 29509 raft_consensus.cc:1905] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W0109 07:20:34.110554 29369 leader_election.cc:271] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer c5a3e756029444b2bf66ee0044f5bc35: Timed out: RequestConsensusVote RPC to 172.31.112.255:7050 timed out after 0.444s (SENT)
I0109 07:20:36.685364 29447 raft_consensus.cc:884] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [term 4 FOLLOWER]: Rejecting Update request from peer 710c236995db46699eb9055c19ac506e for earlier term 2. Current term is 4. Ops: []
I0109 07:20:37.429045 29414 raft_consensus.cc:410] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1678276a36f849c28459de6b6788965a)
I0109 07:20:37.429118 29414 raft_consensus.cc:2158] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [term 4 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 0.313s
I0109 07:20:37.429184 29414 raft_consensus.cc:434] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "710c236995db46699eb9055c19ac506e" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-4.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "1678276a36f849c28459de6b6788965a" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-5.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "591b1a1d383d4b659d7549653f4f136e" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-2.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "c5a3e756029444b2bf66ee0044f5bc35" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-1.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "7897578c54854ba0a5af32eb2aa04704" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-3.gce.cloudera.com" port: 7050 } }
I0109 07:20:43.632985 29530 tablet_server_main.cc:56] Tablet server non-default flags:
--raft_heartbeat_interval_ms=50
--fs_data_dirs=/var/lib/kudu/tserver
--fs_wal_dir=/var/lib/kudu/tserver
--tserver_master_addrs=aserbin-kudu-jepsen-master.gce.cloudera.com:7051
--heap_profile_path=/tmp/kudu-tserver.29530
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--flagfile=/etc/kudu/conf/tserver.gflagfile
--log_dir=/var/log/kudu
--logtostderr=true
Tablet server version:
kudu 1.3.0-SNAPSHOT
revision 3a23b058af0f4de04f8827255de405fff7bf9ee1
build type RELEASE
built by jenkins at 09 Jan 2017 06:07:18 PST on aserbin-kudu-jepsen-control-node-debian8.gce.cloudera.com
build id 2017-01-09_06-07-06
I0109 07:20:43.633153 29530 mem_tracker.cc:140] MemTracker: hard memory limit is 0.464404 GB
I0109 07:20:43.633164 29530 mem_tracker.cc:142] MemTracker: soft memory limit is 0.278643 GB
I0109 07:20:43.634373 29530 tablet_server_main.cc:63] Initializing tablet server...
I0109 07:20:43.684483 29530 hybrid_clock.cc:177] HybridClock initialized. Resolution in nanos?: 1 Wait times tolerance adjustment: 1.0005 Current error: 916540
I0109 07:20:43.685277 29530 env_posix.cc:1286] Not raising process file limit of 65536; it is already as high as it can go
I0109 07:20:43.685309 29530 file_cache.cc:401] Constructed file cache lbm with capacity 32768
I0109 07:20:43.686338 29530 fs_manager.cc:251] Opened local filesystem: /var/lib/kudu/tserver
uuid: "591b1a1d383d4b659d7549653f4f136e"
format_stamp: "Formatted at 2017-01-09 15:20:10 on aserbin-kudu-jepsen-ts-2"
I0109 07:20:43.689476 29530 ts_tablet_manager.cc:701] Loading metadata for tablet 9e99b710634a44f4a948132a5a180600
I0109 07:20:43.689725 29530 ts_tablet_manager.cc:863] Registered tablet 9e99b710634a44f4a948132a5a180600
I0109 07:20:43.689885 29544 ts_tablet_manager.cc:725] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e: Bootstrapping tablet
I0109 07:20:43.690079 29544 tablet_bootstrap.cc:381] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e: Bootstrap starting.
I0109 07:20:43.690331 29544 tablet_bootstrap.cc:540] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e: Time spent opening tablet: real 0.000s	user 0.000s	sys 0.000s
I0109 07:20:43.690433 29544 tablet_bootstrap.cc:596] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e: Will attempt to recover log segment /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600/wal-000000001 to /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery/wal-000000001
I0109 07:20:43.690445 29544 tablet_bootstrap.cc:604] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e: Moving log directory /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600 to recovery directory /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery in preparation for log replay
I0109 07:20:43.690615 29544 log_util.cc:315] Log segment /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery/wal-000000001 has no footer. This segment was likely being written when the server previously shut down.
I0109 07:20:43.690625 29544 log_reader.cc:160] Log segment /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery/wal-000000001 was likely left in-progress after a previous crash. Will try to rebuild footer by scanning data.
I0109 07:20:43.692191 29530 tablet_server_main.cc:66] Starting tablet server...
I0109 07:20:43.693543 29544 log_util.cc:569] Scanning /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery/wal-000000001 for valid entry headers following offset 13788...
I0109 07:20:43.760820 29544 log_util.cc:606] Found no log entry headers
I0109 07:20:43.760960 29544 log_util.cc:218] Ignoring log segment corruption in /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery/wal-000000001 because there are no log entries following the corrupted one. The server probably crashed in the middle of writing an entry to the write-ahead log or downloaded an active log via tablet copy. Error detail: Corruption: CRC mismatch in log entry header: Log file corruption detected. Failed trying to read batch #0 at offset 13776 for log segment /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery/wal-000000001: Prior entries: [off=13511 REPLICATE (4.60)] [off=13545 COMMIT (4.60)] [off=13742 REPLICATE (4.61)] [off=13776 COMMIT (4.61)]
I0109 07:20:43.760980 29544 log_util.cc:368] Successfully rebuilt footer for segment: /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery/wal-000000001 (valid entries through byte offset 13776)
I0109 07:20:43.761049 29544 tablet.cc:983] T 9e99b710634a44f4a948132a5a180600 Rewinding schema during bootstrap to Schema [
	0:key[string NOT NULL],
	1:value[int32 NOT NULL]
]
I0109 07:20:43.761324 29544 log.cc:351] Log is configured to *not* fsync() on all Append() calls
I0109 07:20:43.765453 29544 tablet_bootstrap.cc:381] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e: Bootstrap replayed 1/1 log segments. Stats: ops{read=61 overwritten=0 applied=61 ignored=0} inserts{seen=59 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I0109 07:20:43.765478 29544 tablet_bootstrap.cc:1017] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e: ReplayState: Previous OpId: 4.61, Committed OpId: 4.61, Pending Replicates: 0, Pending Commits: 0
I0109 07:20:43.766199 29544 tablet_bootstrap.cc:633] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e: Preparing to delete log recovery files and directory /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery
I0109 07:20:43.766274 29544 tablet_bootstrap.cc:636] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e: Renaming log recovery dir from /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery to /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery-1483975243766270
I0109 07:20:43.766314 29544 tablet_bootstrap.cc:646] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e: Deleting all files from renamed log recovery directory /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery-1483975243766270
I0109 07:20:43.766446 29544 tablet_bootstrap.cc:649] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e: Completed deletion of old log recovery files and directory /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery-1483975243766270
I0109 07:20:43.766458 29544 tablet_bootstrap.cc:381] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e: Bootstrap complete.
I0109 07:20:43.773864 29544 ts_tablet_manager.cc:730] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e: Time spent bootstrapping tablet: real 0.084s	user 0.016s	sys 0.028s
I0109 07:20:43.774377 29544 raft_consensus.cc:287] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [term 4 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "710c236995db46699eb9055c19ac506e" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-4.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "1678276a36f849c28459de6b6788965a" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-5.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "591b1a1d383d4b659d7549653f4f136e" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-2.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "c5a3e756029444b2bf66ee0044f5bc35" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-1.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "7897578c54854ba0a5af32eb2aa04704" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-3.gce.cloudera.com" port: 7050 } }
I0109 07:20:43.774406 29544 raft_consensus.cc:547] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [term 4 FOLLOWER]: Becoming Follower/Learner. State: Replica: 591b1a1d383d4b659d7549653f4f136e, State: 1, Role: FOLLOWER
I0109 07:20:43.774420 29544 consensus_queue.cc:175] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 61, Last appended: 4.61, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER
I0109 07:20:43.774602 29544 ts_tablet_manager.cc:757] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e: Time spent starting tablet: real 0.001s	user 0.000s	sys 0.000s
I0109 07:20:43.782438 29530 rpc_server.cc:164] RPC server started. Bound to: 0.0.0.0:7050
I0109 07:20:43.782519 29530 webserver.cc:126] Starting webserver on 0.0.0.0:8050
I0109 07:20:43.782528 29530 webserver.cc:137] Document root disabled
I0109 07:20:43.782721 29530 webserver.cc:221] Webserver started. Bound to: http://0.0.0.0:8050/
I0109 07:20:43.783107 29530 tablet_server_main.cc:69] Tablet server successfully started.
I0109 07:20:43.840617 29660 heartbeater.cc:292] Connected to a master server at aserbin-kudu-jepsen-master.gce.cloudera.com:7051
I0109 07:20:43.840664 29660 heartbeater.cc:359] Registering TS with master...
I0109 07:20:43.840807 29660 heartbeater.cc:373] Master aserbin-kudu-jepsen-master.gce.cloudera.com:7051 requested a full tablet report, sending...
I0109 07:20:43.981036 29583 raft_consensus.cc:410] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0109 07:20:43.981123 29583 raft_consensus.cc:2158] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [term 4 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 0.215s
I0109 07:20:43.981206 29583 raft_consensus.cc:434] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "710c236995db46699eb9055c19ac506e" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-4.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "1678276a36f849c28459de6b6788965a" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-5.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "591b1a1d383d4b659d7549653f4f136e" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-2.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "c5a3e756029444b2bf66ee0044f5bc35" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-1.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "7897578c54854ba0a5af32eb2aa04704" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-3.gce.cloudera.com" port: 7050 } }
I0109 07:20:44.179824 29583 leader_election.cc:215] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [CANDIDATE]: Term 5 pre-election: Requesting pre-vote from peer 710c236995db46699eb9055c19ac506e
I0109 07:20:44.180078 29583 leader_election.cc:215] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [CANDIDATE]: Term 5 pre-election: Requesting pre-vote from peer 1678276a36f849c28459de6b6788965a
I0109 07:20:44.180203 29583 leader_election.cc:215] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [CANDIDATE]: Term 5 pre-election: Requesting pre-vote from peer c5a3e756029444b2bf66ee0044f5bc35
I0109 07:20:44.180289 29583 leader_election.cc:215] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [CANDIDATE]: Term 5 pre-election: Requesting pre-vote from peer 7897578c54854ba0a5af32eb2aa04704
W0109 07:20:44.181008 29541 leader_election.cc:271] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 710c236995db46699eb9055c19ac506e: Network error: Client connection negotiation failed: client connection to 172.31.114.38:7050: connect: Connection refused (error 111)
W0109 07:20:44.181331 29540 leader_election.cc:271] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 1678276a36f849c28459de6b6788965a: Network error: Client connection negotiation failed: client connection to 172.31.114.39:7050: connect: Connection refused (error 111)
W0109 07:20:44.181382 29542 leader_election.cc:271] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 7897578c54854ba0a5af32eb2aa04704: Network error: Client connection negotiation failed: client connection to 172.31.114.37:7050: connect: Connection refused (error 111)
I0109 07:20:44.181393 29542 leader_election.cc:242] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost.
I0109 07:20:44.181548 29666 raft_consensus.cc:2158] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [term 4 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 0.176s
I0109 07:20:44.181565 29666 raft_consensus.cc:1905] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I0109 07:20:44.182471 29539 leader_election.cc:347] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [CANDIDATE]: Term 5 pre-election: Vote granted by peer c5a3e756029444b2bf66ee0044f5bc35
I0109 07:20:44.507603 29583 raft_consensus.cc:410] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0109 07:20:44.507686 29583 raft_consensus.cc:2158] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [term 4 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 0.198s
I0109 07:20:44.507753 29583 raft_consensus.cc:434] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "710c236995db46699eb9055c19ac506e" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-4.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "1678276a36f849c28459de6b6788965a" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-5.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "591b1a1d383d4b659d7549653f4f136e" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-2.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "c5a3e756029444b2bf66ee0044f5bc35" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-1.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "7897578c54854ba0a5af32eb2aa04704" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-3.gce.cloudera.com" port: 7050 } }
I0109 07:20:44.704118 29617 raft_consensus.cc:2158] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [term 4 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 0.225s
I0109 07:20:44.704174 29617 raft_consensus.cc:1744] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 710c236995db46699eb9055c19ac506e in term 4.
I0109 07:20:44.704411 29583 leader_election.cc:215] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [CANDIDATE]: Term 5 pre-election: Requesting pre-vote from peer 710c236995db46699eb9055c19ac506e
I0109 07:20:44.704728 29583 leader_election.cc:215] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [CANDIDATE]: Term 5 pre-election: Requesting pre-vote from peer 1678276a36f849c28459de6b6788965a
I0109 07:20:44.704876 29583 leader_election.cc:215] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [CANDIDATE]: Term 5 pre-election: Requesting pre-vote from peer c5a3e756029444b2bf66ee0044f5bc35
I0109 07:20:44.704931 29583 leader_election.cc:215] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [CANDIDATE]: Term 5 pre-election: Requesting pre-vote from peer 7897578c54854ba0a5af32eb2aa04704
I0109 07:20:44.706470 29540 leader_election.cc:347] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [CANDIDATE]: Term 5 pre-election: Vote granted by peer 1678276a36f849c28459de6b6788965a
I0109 07:20:44.706544 29542 leader_election.cc:347] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [CANDIDATE]: Term 5 pre-election: Vote granted by peer 7897578c54854ba0a5af32eb2aa04704
I0109 07:20:44.706557 29542 leader_election.cc:242] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won.
I0109 07:20:44.706684 29676 raft_consensus.cc:2158] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [term 4 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 0.217s
I0109 07:20:44.706702 29676 raft_consensus.cc:1968] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [term 4 FOLLOWER]: Leader pre-election won for term 5
I0109 07:20:44.706708 29676 raft_consensus.cc:410] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I0109 07:20:44.706718 29676 raft_consensus.cc:2158] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [term 4 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 0.179s
I0109 07:20:44.706724 29676 raft_consensus.cc:2202] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [term 4 FOLLOWER]: Advancing to term 5
I0109 07:20:44.707118 29676 raft_consensus.cc:434] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "710c236995db46699eb9055c19ac506e" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-4.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "1678276a36f849c28459de6b6788965a" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-5.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "591b1a1d383d4b659d7549653f4f136e" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-2.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "c5a3e756029444b2bf66ee0044f5bc35" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-1.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "7897578c54854ba0a5af32eb2aa04704" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-3.gce.cloudera.com" port: 7050 } }
I0109 07:20:44.736623 29539 leader_election.cc:361] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [CANDIDATE]: Term 5 pre-election: Vote denied by peer c5a3e756029444b2bf66ee0044f5bc35. Message: Service unavailable: T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 591b1a1d383d4b659d7549653f4f136e for term 5 because replica is already servicing an update from a current leader or another vote.
I0109 07:20:44.842250 29660 heartbeater.cc:365] Master aserbin-kudu-jepsen-master.gce.cloudera.com:7051 was elected leader, sending a full tablet report...
I0109 07:20:44.902997 29617 raft_consensus.cc:1674] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c5a3e756029444b2bf66ee0044f5bc35 in current term 5: Already voted for candidate 591b1a1d383d4b659d7549653f4f136e in this term.
W0109 07:20:44.903126 29617 rpcz_store.cc:234] Call kudu.consensus.ConsensusService.RequestConsensusVote from 172.31.112.255:41908 (request call id 1) took 166ms (client timeout 201).
W0109 07:20:44.903177 29617 rpcz_store.cc:238] Trace:
0109 07:20:44.737070 (+     0us) service_pool.cc:143] Inserting onto call queue
0109 07:20:44.737080 (+    10us) service_pool.cc:202] Handling call
0109 07:20:44.903097 (+166017us) spinlock_profiling.cc:255] Waited 166 ms on lock 0x3ee6ee0. stack: 0000000001a20b21 00000000009d8b77 00000000007e48af 0000000000a8971f 0000000000a8a342 0000000001a2dc19 00007fe00dade0a3 00007fe00c36e87c ffffffffffffffff
0109 07:20:44.903111 (+    14us) inbound_call.cc:130] Queueing success response
Metrics: {"spinlock_wait_cycles":414651136}
W0109 07:20:44.903450 29541 leader_election.cc:271] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 710c236995db46699eb9055c19ac506e: Timed out: RequestConsensusVote RPC to 172.31.114.38:7050 timed out after 0.198s (SENT)
I0109 07:20:44.903550 29617 raft_consensus.cc:1674] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 710c236995db46699eb9055c19ac506e in current term 5: Already voted for candidate 591b1a1d383d4b659d7549653f4f136e in this term.
I0109 07:20:44.903602 29676 leader_election.cc:215] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [CANDIDATE]: Term 5 election: Requesting vote from peer 710c236995db46699eb9055c19ac506e
I0109 07:20:44.903690 29676 leader_election.cc:215] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [CANDIDATE]: Term 5 election: Requesting vote from peer 1678276a36f849c28459de6b6788965a
I0109 07:20:44.903750 29676 leader_election.cc:215] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [CANDIDATE]: Term 5 election: Requesting vote from peer c5a3e756029444b2bf66ee0044f5bc35
I0109 07:20:44.903792 29676 leader_election.cc:215] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [CANDIDATE]: Term 5 election: Requesting vote from peer 7897578c54854ba0a5af32eb2aa04704
I0109 07:20:44.903872 29541 leader_election.cc:361] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [CANDIDATE]: Term 5 election: Vote denied by peer 710c236995db46699eb9055c19ac506e. Message: Invalid argument: T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 591b1a1d383d4b659d7549653f4f136e in current term 5: Already voted for candidate 710c236995db46699eb9055c19ac506e in this term.
I0109 07:20:44.904551 29542 leader_election.cc:347] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [CANDIDATE]: Term 5 election: Vote granted by peer 7897578c54854ba0a5af32eb2aa04704
I0109 07:20:44.904691 29539 leader_election.cc:361] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [CANDIDATE]: Term 5 election: Vote denied by peer c5a3e756029444b2bf66ee0044f5bc35. Message: Invalid argument: T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 591b1a1d383d4b659d7549653f4f136e in current term 5: Already voted for candidate 710c236995db46699eb9055c19ac506e in this term.
I0109 07:20:44.904995 29540 leader_election.cc:361] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [CANDIDATE]: Term 5 election: Vote denied by peer 1678276a36f849c28459de6b6788965a. Message: Invalid argument: T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 591b1a1d383d4b659d7549653f4f136e in current term 5: Already voted for candidate 710c236995db46699eb9055c19ac506e in this term.
I0109 07:20:44.905007 29540 leader_election.cc:242] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost.
I0109 07:20:44.905046 29676 raft_consensus.cc:2158] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [term 5 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 0.298s
I0109 07:20:44.905058 29676 raft_consensus.cc:1905] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I0109 07:20:45.103729 29617 raft_consensus.cc:915] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [term 5 FOLLOWER]: Refusing update from remote peer 710c236995db46699eb9055c19ac506e: Log matching property violated. Preceding OpId in replica: term: 4 index: 61. Preceding OpId from leader: term: 5 index: 62. (index mismatch)
I0109 07:21:07.338804 29713 tablet_server_main.cc:56] Tablet server non-default flags:
--raft_heartbeat_interval_ms=50
--fs_data_dirs=/var/lib/kudu/tserver
--fs_wal_dir=/var/lib/kudu/tserver
--tserver_master_addrs=aserbin-kudu-jepsen-master.gce.cloudera.com:7051
--heap_profile_path=/tmp/kudu-tserver.29713
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--flagfile=/etc/kudu/conf/tserver.gflagfile
--log_dir=/var/log/kudu
--logtostderr=true
Tablet server version:
kudu 1.3.0-SNAPSHOT
revision 3a23b058af0f4de04f8827255de405fff7bf9ee1
build type RELEASE
built by jenkins at 09 Jan 2017 06:07:18 PST on aserbin-kudu-jepsen-control-node-debian8.gce.cloudera.com
build id 2017-01-09_06-07-06
I0109 07:21:07.338968 29713 mem_tracker.cc:140] MemTracker: hard memory limit is 0.464404 GB
I0109 07:21:07.338984 29713 mem_tracker.cc:142] MemTracker: soft memory limit is 0.278643 GB
I0109 07:21:07.340387 29713 tablet_server_main.cc:63] Initializing tablet server...
I0109 07:21:07.390039 29713 hybrid_clock.cc:177] HybridClock initialized. Resolution in nanos?: 1 Wait times tolerance adjustment: 1.0005 Current error: 928540
I0109 07:21:07.390858 29713 env_posix.cc:1286] Not raising process file limit of 65536; it is already as high as it can go
I0109 07:21:07.390889 29713 file_cache.cc:401] Constructed file cache lbm with capacity 32768
I0109 07:21:07.391923 29713 fs_manager.cc:251] Opened local filesystem: /var/lib/kudu/tserver
uuid: "591b1a1d383d4b659d7549653f4f136e"
format_stamp: "Formatted at 2017-01-09 15:20:10 on aserbin-kudu-jepsen-ts-2"
I0109 07:21:07.395262 29713 ts_tablet_manager.cc:701] Loading metadata for tablet 9e99b710634a44f4a948132a5a180600
I0109 07:21:07.395571 29713 ts_tablet_manager.cc:863] Registered tablet 9e99b710634a44f4a948132a5a180600
I0109 07:21:07.395740 29727 ts_tablet_manager.cc:725] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e: Bootstrapping tablet
I0109 07:21:07.395927 29727 tablet_bootstrap.cc:381] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e: Bootstrap starting.
I0109 07:21:07.396216 29727 tablet_bootstrap.cc:540] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e: Time spent opening tablet: real 0.000s	user 0.000s	sys 0.000s
I0109 07:21:07.396294 29727 tablet_bootstrap.cc:596] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e: Will attempt to recover log segment /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600/wal-000000001 to /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery/wal-000000001
I0109 07:21:07.396322 29727 tablet_bootstrap.cc:604] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e: Moving log directory /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600 to recovery directory /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery in preparation for log replay
I0109 07:21:07.396553 29727 log_util.cc:315] Log segment /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery/wal-000000001 has no footer. This segment was likely being written when the server previously shut down.
I0109 07:21:07.396589 29727 log_reader.cc:160] Log segment /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery/wal-000000001 was likely left in-progress after a previous crash. Will try to rebuild footer by scanning data.
I0109 07:21:07.398061 29713 tablet_server_main.cc:66] Starting tablet server...
I0109 07:21:07.400012 29727 log_util.cc:569] Scanning /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery/wal-000000001 for valid entry headers following offset 36592...
I0109 07:21:07.485064 29727 log_util.cc:606] Found no log entry headers
I0109 07:21:07.485137 29727 log_util.cc:218] Ignoring log segment corruption in /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery/wal-000000001 because there are no log entries following the corrupted one. The server probably crashed in the middle of writing an entry to the write-ahead log or downloaded an active log via tablet copy. Error detail: Corruption: CRC mismatch in log entry header: Log file corruption detected. Failed trying to read batch #0 at offset 36580 for log segment /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery/wal-000000001: Prior entries: [off=36312 REPLICATE (5.159)] [off=36347 COMMIT (5.159)] [off=36545 REPLICATE (5.160)] [off=36580 COMMIT (5.160)]
I0109 07:21:07.485155 29727 log_util.cc:368] Successfully rebuilt footer for segment: /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery/wal-000000001 (valid entries through byte offset 36580)
I0109 07:21:07.485261 29727 tablet.cc:983] T 9e99b710634a44f4a948132a5a180600 Rewinding schema during bootstrap to Schema [
	0:key[string NOT NULL],
	1:value[int32 NOT NULL]
]
I0109 07:21:07.485594 29727 log.cc:351] Log is configured to *not* fsync() on all Append() calls
I0109 07:21:07.505926 29727 tablet_bootstrap.cc:381] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e: Bootstrap replayed 1/1 log segments. Stats: ops{read=160 overwritten=0 applied=160 ignored=0} inserts{seen=157 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I0109 07:21:07.505990 29727 tablet_bootstrap.cc:1017] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e: ReplayState: Previous OpId: 5.160, Committed OpId: 5.160, Pending Replicates: 0, Pending Commits: 0
I0109 07:21:07.506649 29727 tablet_bootstrap.cc:633] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e: Preparing to delete log recovery files and directory /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery
I0109 07:21:07.506703 29727 tablet_bootstrap.cc:636] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e: Renaming log recovery dir from /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery to /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery-1483975267506700
I0109 07:21:07.506758 29727 tablet_bootstrap.cc:646] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e: Deleting all files from renamed log recovery directory /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery-1483975267506700
I0109 07:21:07.506983 29727 tablet_bootstrap.cc:649] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e: Completed deletion of old log recovery files and directory /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery-1483975267506700
I0109 07:21:07.507020 29727 tablet_bootstrap.cc:381] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e: Bootstrap complete.
I0109 07:21:07.515557 29727 ts_tablet_manager.cc:730] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e: Time spent bootstrapping tablet: real 0.120s	user 0.024s	sys 0.040s
I0109 07:21:07.516511 29727 raft_consensus.cc:287] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [term 5 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "710c236995db46699eb9055c19ac506e" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-4.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "1678276a36f849c28459de6b6788965a" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-5.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "591b1a1d383d4b659d7549653f4f136e" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-2.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "c5a3e756029444b2bf66ee0044f5bc35" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-1.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "7897578c54854ba0a5af32eb2aa04704" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-3.gce.cloudera.com" port: 7050 } }
I0109 07:21:07.516554 29727 raft_consensus.cc:547] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [term 5 FOLLOWER]: Becoming Follower/Learner. State: Replica: 591b1a1d383d4b659d7549653f4f136e, State: 1, Role: FOLLOWER
I0109 07:21:07.516580 29727 consensus_queue.cc:175] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 160, Last appended: 5.160, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER
I0109 07:21:07.516801 29727 ts_tablet_manager.cc:757] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e: Time spent starting tablet: real 0.001s	user 0.000s	sys 0.000s
I0109 07:21:07.523469 29713 rpc_server.cc:164] RPC server started. Bound to: 0.0.0.0:7050
I0109 07:21:07.523551 29713 webserver.cc:126] Starting webserver on 0.0.0.0:8050
I0109 07:21:07.523558 29713 webserver.cc:137] Document root disabled
I0109 07:21:07.523712 29713 webserver.cc:221] Webserver started. Bound to: http://0.0.0.0:8050/
I0109 07:21:07.524133 29713 tablet_server_main.cc:69] Tablet server successfully started.
I0109 07:21:07.574290 29800 raft_consensus.cc:2202] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [term 5 FOLLOWER]: Advancing to term 6
I0109 07:21:07.574724 29800 raft_consensus.cc:915] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [term 6 FOLLOWER]: Refusing update from remote peer 710c236995db46699eb9055c19ac506e: Log matching property violated. Preceding OpId in replica: term: 5 index: 160. Preceding OpId from leader: term: 6 index: 162. (index mismatch)
I0109 07:21:07.575999 29843 heartbeater.cc:292] Connected to a master server at aserbin-kudu-jepsen-master.gce.cloudera.com:7051
I0109 07:21:07.576040 29843 heartbeater.cc:359] Registering TS with master...
I0109 07:21:07.576114 29843 heartbeater.cc:373] Master aserbin-kudu-jepsen-master.gce.cloudera.com:7051 requested a full tablet report, sending...
I0109 07:21:07.576817 29843 heartbeater.cc:365] Master aserbin-kudu-jepsen-master.gce.cloudera.com:7051 was elected leader, sending a full tablet report...
I0109 07:21:25.434722 29871 tablet_server_main.cc:56] Tablet server non-default flags:
--raft_heartbeat_interval_ms=50
--fs_data_dirs=/var/lib/kudu/tserver
--fs_wal_dir=/var/lib/kudu/tserver
--tserver_master_addrs=aserbin-kudu-jepsen-master.gce.cloudera.com:7051
--heap_profile_path=/tmp/kudu-tserver.29871
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--flagfile=/etc/kudu/conf/tserver.gflagfile
--log_dir=/var/log/kudu
--logtostderr=true
Tablet server version:
kudu 1.3.0-SNAPSHOT
revision 3a23b058af0f4de04f8827255de405fff7bf9ee1
build type RELEASE
built by jenkins at 09 Jan 2017 06:07:18 PST on aserbin-kudu-jepsen-control-node-debian8.gce.cloudera.com
build id 2017-01-09_06-07-06
I0109 07:21:25.434888 29871 mem_tracker.cc:140] MemTracker: hard memory limit is 0.464404 GB
I0109 07:21:25.434900 29871 mem_tracker.cc:142] MemTracker: soft memory limit is 0.278643 GB
I0109 07:21:25.436105 29871 tablet_server_main.cc:63] Initializing tablet server...
I0109 07:21:25.485257 29871 hybrid_clock.cc:177] HybridClock initialized. Resolution in nanos?: 1 Wait times tolerance adjustment: 1.0005 Current error: 937540
I0109 07:21:25.486083 29871 env_posix.cc:1286] Not raising process file limit of 65536; it is already as high as it can go
I0109 07:21:25.486119 29871 file_cache.cc:401] Constructed file cache lbm with capacity 32768
I0109 07:21:25.487257 29871 fs_manager.cc:251] Opened local filesystem: /var/lib/kudu/tserver
uuid: "591b1a1d383d4b659d7549653f4f136e"
format_stamp: "Formatted at 2017-01-09 15:20:10 on aserbin-kudu-jepsen-ts-2"
I0109 07:21:25.490545 29871 ts_tablet_manager.cc:701] Loading metadata for tablet 9e99b710634a44f4a948132a5a180600
I0109 07:21:25.490825 29871 ts_tablet_manager.cc:863] Registered tablet 9e99b710634a44f4a948132a5a180600
I0109 07:21:25.490983 29885 ts_tablet_manager.cc:725] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e: Bootstrapping tablet
I0109 07:21:25.491197 29885 tablet_bootstrap.cc:381] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e: Bootstrap starting.
I0109 07:21:25.491451 29885 tablet_bootstrap.cc:540] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e: Time spent opening tablet: real 0.000s	user 0.000s	sys 0.000s
I0109 07:21:25.491523 29885 tablet_bootstrap.cc:596] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e: Will attempt to recover log segment /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600/wal-000000001 to /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery/wal-000000001
I0109 07:21:25.491535 29885 tablet_bootstrap.cc:604] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e: Moving log directory /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600 to recovery directory /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery in preparation for log replay
I0109 07:21:25.491740 29885 log_util.cc:315] Log segment /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery/wal-000000001 has no footer. This segment was likely being written when the server previously shut down.
I0109 07:21:25.491750 29885 log_reader.cc:160] Log segment /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery/wal-000000001 was likely left in-progress after a previous crash. Will try to rebuild footer by scanning data.
I0109 07:21:25.493257 29871 tablet_server_main.cc:66] Starting tablet server...
I0109 07:21:25.495798 29885 log_util.cc:569] Scanning /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery/wal-000000001 for valid entry headers following offset 48028...
I0109 07:21:25.566491 29885 log_util.cc:606] Found no log entry headers
I0109 07:21:25.566596 29885 log_util.cc:218] Ignoring log segment corruption in /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery/wal-000000001 because there are no log entries following the corrupted one. The server probably crashed in the middle of writing an entry to the write-ahead log or downloaded an active log via tablet copy. Error detail: Corruption: CRC mismatch in log entry header: Log file corruption detected. Failed trying to read batch #0 at offset 48016 for log segment /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery/wal-000000001: Prior entries: [off=47585 COMMIT (6.208)] [off=47783 REPLICATE (6.209)] [off=47818 COMMIT (6.209)] [off=48016 REPLICATE (6.210)]
I0109 07:21:25.566611 29885 log_util.cc:368] Successfully rebuilt footer for segment: /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery/wal-000000001 (valid entries through byte offset 48016)
I0109 07:21:25.566673 29885 tablet.cc:983] T 9e99b710634a44f4a948132a5a180600 Rewinding schema during bootstrap to Schema [
	0:key[string NOT NULL],
	1:value[int32 NOT NULL]
]
I0109 07:21:25.566975 29885 log.cc:351] Log is configured to *not* fsync() on all Append() calls
I0109 07:21:25.581609 29885 tablet_bootstrap.cc:381] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e: Bootstrap replayed 1/1 log segments. Stats: ops{read=210 overwritten=0 applied=209 ignored=0} inserts{seen=205 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I0109 07:21:25.581792 29885 tablet_bootstrap.cc:1017] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e: ReplayState: Previous OpId: 6.210, Committed OpId: 6.209, Pending Replicates: 1, Pending Commits: 0
I0109 07:21:25.581804 29885 tablet_bootstrap.cc:1017] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e: Dumping REPLICATES: 
I0109 07:21:25.581809 29885 tablet_bootstrap.cc:1017] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e:    type: REPLICATE replicate { id { term: 6 index: 210 } timestamp: 6078362722740658176 op_type: WRITE_OP write_request { tablet_id: "9e99b710634a44f4a948132a5a180600" schema { columns { name: "key" type: STRING is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "value" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } row_operations { rows: "\005\003\000\000\000\000\000\000\000\000\001\000\000\000\000\000\000\000\004\000\000\000" indirect_data: "x" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 6078362720224337920 } request_id { client_id: "551b04fdc262441283af896460477398" seq_no: 41 first_incomplete_seq_no: 41 attempt_no: 1 } }
I0109 07:21:25.582262 29885 tablet_bootstrap.cc:633] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e: Preparing to delete log recovery files and directory /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery
I0109 07:21:25.582293 29885 tablet_bootstrap.cc:636] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e: Renaming log recovery dir from /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery to /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery-1483975285582292
I0109 07:21:25.582314 29885 tablet_bootstrap.cc:646] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e: Deleting all files from renamed log recovery directory /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery-1483975285582292
I0109 07:21:25.582482 29885 tablet_bootstrap.cc:649] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e: Completed deletion of old log recovery files and directory /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery-1483975285582292
I0109 07:21:25.582494 29885 tablet_bootstrap.cc:381] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e: Bootstrap complete.
I0109 07:21:25.587955 29885 ts_tablet_manager.cc:730] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e: Time spent bootstrapping tablet: real 0.097s	user 0.016s	sys 0.040s
I0109 07:21:25.588529 29885 raft_consensus.cc:287] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [term 6 FOLLOWER]: Replica starting. Triggering 1 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "710c236995db46699eb9055c19ac506e" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-4.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "1678276a36f849c28459de6b6788965a" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-5.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "591b1a1d383d4b659d7549653f4f136e" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-2.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "c5a3e756029444b2bf66ee0044f5bc35" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-1.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "7897578c54854ba0a5af32eb2aa04704" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-3.gce.cloudera.com" port: 7050 } }
I0109 07:21:25.588723 29885 raft_consensus.cc:547] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [term 6 FOLLOWER]: Becoming Follower/Learner. State: Replica: 591b1a1d383d4b659d7549653f4f136e, State: 1, Role: FOLLOWER
I0109 07:21:25.588742 29885 consensus_queue.cc:175] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 209, Last appended: 6.210, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER
I0109 07:21:25.588989 29885 ts_tablet_manager.cc:757] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e: Time spent starting tablet: real 0.001s	user 0.000s	sys 0.000s
I0109 07:21:25.595695 29871 rpc_server.cc:164] RPC server started. Bound to: 0.0.0.0:7050
I0109 07:21:25.595784 29871 webserver.cc:126] Starting webserver on 0.0.0.0:8050
I0109 07:21:25.595791 29871 webserver.cc:137] Document root disabled
I0109 07:21:25.595979 29871 webserver.cc:221] Webserver started. Bound to: http://0.0.0.0:8050/
I0109 07:21:25.596405 29871 tablet_server_main.cc:69] Tablet server successfully started.
I0109 07:21:25.648402 30002 heartbeater.cc:292] Connected to a master server at aserbin-kudu-jepsen-master.gce.cloudera.com:7051
I0109 07:21:25.648449 30002 heartbeater.cc:359] Registering TS with master...
I0109 07:21:25.648527 30002 heartbeater.cc:373] Master aserbin-kudu-jepsen-master.gce.cloudera.com:7051 requested a full tablet report, sending...
I0109 07:21:25.745122 29930 raft_consensus.cc:410] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0109 07:21:25.745214 29930 raft_consensus.cc:2158] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [term 6 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 0.213s
I0109 07:21:25.745292 29930 raft_consensus.cc:434] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "710c236995db46699eb9055c19ac506e" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-4.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "1678276a36f849c28459de6b6788965a" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-5.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "591b1a1d383d4b659d7549653f4f136e" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-2.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "c5a3e756029444b2bf66ee0044f5bc35" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-1.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "7897578c54854ba0a5af32eb2aa04704" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-3.gce.cloudera.com" port: 7050 } }
I0109 07:21:25.941777 29930 leader_election.cc:215] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [CANDIDATE]: Term 7 pre-election: Requesting pre-vote from peer 710c236995db46699eb9055c19ac506e
I0109 07:21:25.942052 29959 raft_consensus.cc:1691] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c5a3e756029444b2bf66ee0044f5bc35 for term 7 because replica has last-logged OpId of term: 6 index: 210, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 209.
W0109 07:21:25.942201 29959 rpcz_store.cc:234] Call kudu.consensus.ConsensusService.RequestConsensusVote from 172.31.112.255:41921 (request call id 1) took 131ms (client timeout 150).
W0109 07:21:25.942260 29959 rpcz_store.cc:238] Trace:
0109 07:21:25.810616 (+     0us) service_pool.cc:143] Inserting onto call queue
0109 07:21:25.810626 (+    10us) service_pool.cc:202] Handling call
0109 07:21:25.942172 (+131546us) spinlock_profiling.cc:255] Waited 131 ms on lock 0x45ff300. stack: 0000000001a20b21 00000000009d8b77 00000000007e48af 0000000000a8971f 0000000000a8a342 0000000001a2dc19 00007fc75718d0a3 00007fc755a1d87c ffffffffffffffff
0109 07:21:25.942189 (+    17us) inbound_call.cc:130] Queueing success response
Metrics: {"spinlock_wait_cycles":328354560}
I0109 07:21:25.942334 29930 leader_election.cc:215] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [CANDIDATE]: Term 7 pre-election: Requesting pre-vote from peer 1678276a36f849c28459de6b6788965a
I0109 07:21:25.942451 29930 leader_election.cc:215] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [CANDIDATE]: Term 7 pre-election: Requesting pre-vote from peer c5a3e756029444b2bf66ee0044f5bc35
I0109 07:21:25.942525 29930 leader_election.cc:215] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [CANDIDATE]: Term 7 pre-election: Requesting pre-vote from peer 7897578c54854ba0a5af32eb2aa04704
I0109 07:21:25.944577 29882 leader_election.cc:347] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [CANDIDATE]: Term 7 pre-election: Vote granted by peer 710c236995db46699eb9055c19ac506e
I0109 07:21:25.944777 29881 leader_election.cc:347] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [CANDIDATE]: Term 7 pre-election: Vote granted by peer 1678276a36f849c28459de6b6788965a
I0109 07:21:25.944792 29881 leader_election.cc:242] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won.
I0109 07:21:25.944877 29883 leader_election.cc:347] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [CANDIDATE]: Term 7 pre-election: Vote granted by peer 7897578c54854ba0a5af32eb2aa04704
I0109 07:21:25.945003 30008 raft_consensus.cc:2158] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [term 6 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 0.217s
I0109 07:21:25.945020 30008 raft_consensus.cc:1968] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [term 6 FOLLOWER]: Leader pre-election won for term 7
I0109 07:21:25.945027 30008 raft_consensus.cc:410] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I0109 07:21:25.945036 30008 raft_consensus.cc:2158] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [term 6 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 0.172s
I0109 07:21:25.945042 30008 raft_consensus.cc:2202] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [term 6 FOLLOWER]: Advancing to term 7
I0109 07:21:25.945406 30008 raft_consensus.cc:434] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "710c236995db46699eb9055c19ac506e" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-4.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "1678276a36f849c28459de6b6788965a" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-5.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "591b1a1d383d4b659d7549653f4f136e" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-2.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "c5a3e756029444b2bf66ee0044f5bc35" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-1.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "7897578c54854ba0a5af32eb2aa04704" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-3.gce.cloudera.com" port: 7050 } }
I0109 07:21:26.010327 29880 leader_election.cc:361] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [CANDIDATE]: Term 7 pre-election: Vote denied by peer c5a3e756029444b2bf66ee0044f5bc35. Message: Invalid argument: T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 591b1a1d383d4b659d7549653f4f136e in current term 7: Already voted for candidate c5a3e756029444b2bf66ee0044f5bc35 in this term.
I0109 07:21:26.142189 29959 raft_consensus.cc:1674] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate c5a3e756029444b2bf66ee0044f5bc35 in current term 7: Already voted for candidate 591b1a1d383d4b659d7549653f4f136e in this term.
W0109 07:21:26.142326 29959 rpcz_store.cc:234] Call kudu.consensus.ConsensusService.RequestConsensusVote from 172.31.112.255:41921 (request call id 2) took 132ms (client timeout 165).
W0109 07:21:26.142357 29959 rpcz_store.cc:238] Trace:
0109 07:21:26.009882 (+     0us) service_pool.cc:143] Inserting onto call queue
0109 07:21:26.009908 (+    26us) service_pool.cc:202] Handling call
0109 07:21:26.142299 (+132391us) spinlock_profiling.cc:255] Waited 132 ms on lock 0x45ff300. stack: 0000000001a20b21 00000000009d8b77 00000000007e48af 0000000000a8971f 0000000000a8a342 0000000001a2dc19 00007fc75718d0a3 00007fc755a1d87c ffffffffffffffff
0109 07:21:26.142311 (+    12us) inbound_call.cc:130] Queueing success response
Metrics: {"spinlock_wait_cycles":330529536}
I0109 07:21:26.142467 30008 leader_election.cc:215] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [CANDIDATE]: Term 7 election: Requesting vote from peer 710c236995db46699eb9055c19ac506e
I0109 07:21:26.142536 30008 leader_election.cc:215] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [CANDIDATE]: Term 7 election: Requesting vote from peer 1678276a36f849c28459de6b6788965a
I0109 07:21:26.142583 30008 leader_election.cc:215] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [CANDIDATE]: Term 7 election: Requesting vote from peer c5a3e756029444b2bf66ee0044f5bc35
I0109 07:21:26.142627 30008 leader_election.cc:215] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [CANDIDATE]: Term 7 election: Requesting vote from peer 7897578c54854ba0a5af32eb2aa04704
I0109 07:21:26.143141 29881 leader_election.cc:361] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [CANDIDATE]: Term 7 election: Vote denied by peer 1678276a36f849c28459de6b6788965a. Message: Invalid argument: T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 591b1a1d383d4b659d7549653f4f136e in current term 7: Already voted for candidate c5a3e756029444b2bf66ee0044f5bc35 in this term.
I0109 07:21:26.143234 29883 leader_election.cc:361] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [CANDIDATE]: Term 7 election: Vote denied by peer 7897578c54854ba0a5af32eb2aa04704. Message: Invalid argument: T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 591b1a1d383d4b659d7549653f4f136e in current term 7: Already voted for candidate c5a3e756029444b2bf66ee0044f5bc35 in this term.
I0109 07:21:26.143582 29882 leader_election.cc:347] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [CANDIDATE]: Term 7 election: Vote granted by peer 710c236995db46699eb9055c19ac506e
I0109 07:21:26.210288 29880 leader_election.cc:361] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [CANDIDATE]: Term 7 election: Vote denied by peer c5a3e756029444b2bf66ee0044f5bc35. Message: Invalid argument: T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35 [term 7 LEADER]: Leader election vote request: Denying vote to candidate 591b1a1d383d4b659d7549653f4f136e for term 7 because replica is either leader or believes a valid leader to be alive.
I0109 07:21:26.210340 29880 leader_election.cc:242] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost.
I0109 07:21:26.210464 29959 raft_consensus.cc:915] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [term 7 FOLLOWER]: Refusing update from remote peer c5a3e756029444b2bf66ee0044f5bc35: Log matching property violated. Preceding OpId in replica: term: 6 index: 210. Preceding OpId from leader: term: 7 index: 210. (term mismatch)
I0109 07:21:26.210474 29959 raft_consensus_state.cc:382] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e: Aborting all transactions after (but not including) 209
I0109 07:21:26.210481 29959 raft_consensus_state.cc:403] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e: Aborting uncommitted WRITE_OP operation due to leader change: 6.210
I0109 07:21:26.210597 30008 raft_consensus.cc:2158] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [term 7 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 0.177s
I0109 07:21:26.210609 30008 raft_consensus.cc:1905] T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I0109 07:21:26.211091 30002 heartbeater.cc:365] Master aserbin-kudu-jepsen-master.gce.cloudera.com:7051 was elected leader, sending a full tablet report...