I0109 07:20:11.392645 2130 tablet_server_main.cc:56] Tablet server non-default flags: --raft_heartbeat_interval_ms=50 --fs_data_dirs=/var/lib/kudu/tserver --fs_wal_dir=/var/lib/kudu/tserver --tserver_master_addrs=aserbin-kudu-jepsen-master.gce.cloudera.com:7051 --heap_profile_path=/tmp/kudu-tserver.2130 --unlock_experimental_flags=true --unlock_unsafe_flags=true --flagfile=/etc/kudu/conf/tserver.gflagfile --log_dir=/var/log/kudu --logtostderr=true Tablet server version: kudu 1.3.0-SNAPSHOT revision 3a23b058af0f4de04f8827255de405fff7bf9ee1 build type RELEASE built by jenkins at 09 Jan 2017 06:07:18 PST on aserbin-kudu-jepsen-control-node-debian8.gce.cloudera.com build id 2017-01-09_06-07-06 I0109 07:20:11.395463 2130 mem_tracker.cc:140] MemTracker: hard memory limit is 0.464404 GB I0109 07:20:11.395501 2130 mem_tracker.cc:142] MemTracker: soft memory limit is 0.278643 GB I0109 07:20:11.434849 2130 tablet_server_main.cc:63] Initializing tablet server... I0109 07:20:11.485990 2130 hybrid_clock.cc:177] HybridClock initialized. Resolution in nanos?: 1 Wait times tolerance adjustment: 1.0005 Current error: 588897 I0109 07:20:11.488060 2130 env_posix.cc:1286] Not raising process file limit of 65536; it is already as high as it can go I0109 07:20:11.488107 2130 file_cache.cc:401] Constructed file cache lbm with capacity 32768 I0109 07:20:11.530665 2130 server_base.cc:179] Could not load existing FS layout: Not found: /var/lib/kudu/tserver/instance: No such file or directory (error 2) I0109 07:20:11.530707 2130 server_base.cc:180] Creating new FS layout I0109 07:20:11.542845 2130 fs_manager.cc:362] Generated new instance metadata in path /var/lib/kudu/tserver/instance: uuid: "c5a3e756029444b2bf66ee0044f5bc35" format_stamp: "Formatted at 2017-01-09 15:20:11 on aserbin-kudu-jepsen-ts-1" I0109 07:20:11.553225 2130 fs_manager.cc:251] Opened local filesystem: /var/lib/kudu/tserver uuid: "c5a3e756029444b2bf66ee0044f5bc35" format_stamp: "Formatted at 2017-01-09 15:20:11 on aserbin-kudu-jepsen-ts-1" I0109 07:20:11.559480 2130 tablet_server_main.cc:66] Starting tablet server... I0109 07:20:11.624779 2130 rpc_server.cc:164] RPC server started. Bound to: 0.0.0.0:7050 I0109 07:20:11.624864 2130 webserver.cc:126] Starting webserver on 0.0.0.0:8050 I0109 07:20:11.624871 2130 webserver.cc:137] Document root disabled I0109 07:20:11.625052 2130 webserver.cc:221] Webserver started. Bound to: http://0.0.0.0:8050/ I0109 07:20:11.625557 2130 tablet_server_main.cc:69] Tablet server successfully started. I0109 07:20:11.679226 2274 heartbeater.cc:292] Connected to a master server at aserbin-kudu-jepsen-master.gce.cloudera.com:7051 I0109 07:20:11.679292 2274 heartbeater.cc:359] Registering TS with master... I0109 07:20:11.679391 2274 heartbeater.cc:373] Master aserbin-kudu-jepsen-master.gce.cloudera.com:7051 requested a full tablet report, sending... I0109 07:20:12.680157 2274 heartbeater.cc:365] Master aserbin-kudu-jepsen-master.gce.cloudera.com:7051 was elected leader, sending a full tablet report... I0109 07:20:17.099745 2211 tablet_service.cc:630] Processing CreateTablet for tablet 9e99b710634a44f4a948132a5a180600 (table=rw-register-1483975178870 [id=0e0bbef280744ed1a8036e11eb0aaea0]), partition=RANGE (key) PARTITION UNBOUNDED I0109 07:20:17.107620 2211 ts_tablet_manager.cc:863] Registered tablet 9e99b710634a44f4a948132a5a180600 I0109 07:20:17.107969 2292 ts_tablet_manager.cc:725] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35: Bootstrapping tablet I0109 07:20:17.108176 2292 tablet_bootstrap.cc:381] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35: Bootstrap starting. I0109 07:20:17.109217 2292 tablet_bootstrap.cc:540] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35: Time spent opening tablet: real 0.000s user 0.000s sys 0.000s I0109 07:20:17.109355 2292 tablet_bootstrap.cc:483] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35: No blocks or log segments found. Creating new log. I0109 07:20:17.109560 2292 log.cc:351] Log is configured to *not* fsync() on all Append() calls I0109 07:20:17.110103 2292 tablet_bootstrap.cc:381] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35: No bootstrap required, opened a new log I0109 07:20:17.110153 2292 ts_tablet_manager.cc:730] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.000s I0109 07:20:17.111935 2292 raft_consensus.cc:287] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "710c236995db46699eb9055c19ac506e" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-4.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "1678276a36f849c28459de6b6788965a" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-5.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "591b1a1d383d4b659d7549653f4f136e" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-2.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "c5a3e756029444b2bf66ee0044f5bc35" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-1.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "7897578c54854ba0a5af32eb2aa04704" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-3.gce.cloudera.com" port: 7050 } } I0109 07:20:17.111974 2292 raft_consensus.cc:318] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0109 07:20:17.111984 2292 raft_consensus.cc:547] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5a3e756029444b2bf66ee0044f5bc35, State: 1, Role: FOLLOWER I0109 07:20:17.112000 2292 consensus_queue.cc:175] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER I0109 07:20:17.112185 2292 ts_tablet_manager.cc:757] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s I0109 07:20:17.132863 2295 raft_consensus.cc:410] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0109 07:20:17.132957 2295 raft_consensus.cc:2158] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35 [term 0 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 0.217s I0109 07:20:17.133029 2295 raft_consensus.cc:434] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "710c236995db46699eb9055c19ac506e" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-4.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "1678276a36f849c28459de6b6788965a" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-5.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "591b1a1d383d4b659d7549653f4f136e" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-2.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "c5a3e756029444b2bf66ee0044f5bc35" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-1.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "7897578c54854ba0a5af32eb2aa04704" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-3.gce.cloudera.com" port: 7050 } } I0109 07:20:17.331183 2230 raft_consensus.cc:1719] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35 [term 0 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 591b1a1d383d4b659d7549653f4f136e for term 1 because replica is already servicing an update from a current leader or another vote. I0109 07:20:17.331375 2231 raft_consensus.cc:2158] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35 [term 0 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 0.174s I0109 07:20:17.331425 2231 raft_consensus.cc:1744] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1678276a36f849c28459de6b6788965a in term 0. W0109 07:20:17.331460 2231 rpcz_store.cc:234] Call kudu.consensus.ConsensusService.RequestConsensusVote from 172.31.114.39:44890 (request call id 1) took 181ms (client timeout 172). W0109 07:20:17.331531 2231 rpcz_store.cc:238] Trace: 0109 07:20:17.149491 (+ 0us) service_pool.cc:143] Inserting onto call queue 0109 07:20:17.149526 (+ 35us) service_pool.cc:202] Handling call 0109 07:20:17.331443 (+181917us) spinlock_profiling.cc:255] Waited 182 ms on lock 0x5021cc0. stack: 0109 07:20:17.331447 (+ 4us) inbound_call.cc:130] Queueing success response Metrics: {"spinlock_wait_cycles":454487296} I0109 07:20:17.331655 2295 leader_election.cc:215] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 710c236995db46699eb9055c19ac506e I0109 07:20:17.331907 2295 leader_election.cc:215] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 1678276a36f849c28459de6b6788965a I0109 07:20:17.332330 2295 leader_election.cc:215] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 591b1a1d383d4b659d7549653f4f136e I0109 07:20:17.332435 2295 leader_election.cc:215] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 7897578c54854ba0a5af32eb2aa04704 I0109 07:20:17.334892 2166 leader_election.cc:347] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35 [CANDIDATE]: Term 1 pre-election: Vote granted by peer 1678276a36f849c28459de6b6788965a I0109 07:20:17.422801 2168 leader_election.cc:361] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35 [CANDIDATE]: Term 1 pre-election: Vote denied by peer 7897578c54854ba0a5af32eb2aa04704. Message: Service unavailable: T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [term 0 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c5a3e756029444b2bf66ee0044f5bc35 for term 1 because replica is already servicing an update from a current leader or another vote. I0109 07:20:17.423029 2231 raft_consensus.cc:2158] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35 [term 0 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 0.159s I0109 07:20:17.423049 2231 raft_consensus.cc:1744] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7897578c54854ba0a5af32eb2aa04704 in term 0. I0109 07:20:17.456452 2167 leader_election.cc:361] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35 [CANDIDATE]: Term 1 pre-election: Vote denied by peer 710c236995db46699eb9055c19ac506e. Message: Invalid argument: T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c5a3e756029444b2bf66ee0044f5bc35 in current term 1: Already voted for candidate 710c236995db46699eb9055c19ac506e in this term. I0109 07:20:17.456641 2231 raft_consensus.cc:2202] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35 [term 0 FOLLOWER]: Advancing to term 1 I0109 07:20:17.456699 2231 raft_consensus.cc:2158] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 0.183s I0109 07:20:17.457226 2231 raft_consensus.cc:1744] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 710c236995db46699eb9055c19ac506e in term 1. I0109 07:20:17.457576 2168 leader_election.cc:361] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35 [CANDIDATE]: Term 1 pre-election: Vote denied by peer 591b1a1d383d4b659d7549653f4f136e. Message: Invalid argument: T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c5a3e756029444b2bf66ee0044f5bc35 in current term 1: Already voted for candidate 591b1a1d383d4b659d7549653f4f136e in this term. I0109 07:20:17.457602 2168 leader_election.cc:242] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. I0109 07:20:17.457922 2231 raft_consensus.cc:1674] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 591b1a1d383d4b659d7549653f4f136e in current term 1: Already voted for candidate 710c236995db46699eb9055c19ac506e in this term. I0109 07:20:17.458047 2300 raft_consensus.cc:2158] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 0.218s I0109 07:20:17.458070 2300 raft_consensus.cc:1905] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority I0109 07:20:17.622614 2231 raft_consensus.cc:1674] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 7897578c54854ba0a5af32eb2aa04704 in current term 1: Already voted for candidate 710c236995db46699eb9055c19ac506e in this term. I0109 07:20:17.658622 2231 raft_consensus.cc:915] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35 [term 1 FOLLOWER]: Refusing update from remote peer 710c236995db46699eb9055c19ac506e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch) I0109 07:20:17.805343 2231 raft_consensus.cc:1705] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 591b1a1d383d4b659d7549653f4f136e for term 2 because replica is either leader or believes a valid leader to be alive. I0109 07:20:24.415516 2295 raft_consensus.cc:410] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 710c236995db46699eb9055c19ac506e) I0109 07:20:24.415586 2295 raft_consensus.cc:2158] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 0.156s I0109 07:20:24.415680 2295 raft_consensus.cc:434] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "710c236995db46699eb9055c19ac506e" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-4.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "1678276a36f849c28459de6b6788965a" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-5.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "591b1a1d383d4b659d7549653f4f136e" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-2.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "c5a3e756029444b2bf66ee0044f5bc35" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-1.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "7897578c54854ba0a5af32eb2aa04704" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-3.gce.cloudera.com" port: 7050 } } I0109 07:20:30.838840 2319 tablet_server_main.cc:56] Tablet server non-default flags: --raft_heartbeat_interval_ms=50 --fs_data_dirs=/var/lib/kudu/tserver --fs_wal_dir=/var/lib/kudu/tserver --tserver_master_addrs=aserbin-kudu-jepsen-master.gce.cloudera.com:7051 --heap_profile_path=/tmp/kudu-tserver.2319 --unlock_experimental_flags=true --unlock_unsafe_flags=true --flagfile=/etc/kudu/conf/tserver.gflagfile --log_dir=/var/log/kudu --logtostderr=true Tablet server version: kudu 1.3.0-SNAPSHOT revision 3a23b058af0f4de04f8827255de405fff7bf9ee1 build type RELEASE built by jenkins at 09 Jan 2017 06:07:18 PST on aserbin-kudu-jepsen-control-node-debian8.gce.cloudera.com build id 2017-01-09_06-07-06 I0109 07:20:30.839087 2319 mem_tracker.cc:140] MemTracker: hard memory limit is 0.464404 GB I0109 07:20:30.839105 2319 mem_tracker.cc:142] MemTracker: soft memory limit is 0.278643 GB I0109 07:20:30.840764 2319 tablet_server_main.cc:63] Initializing tablet server... I0109 07:20:30.899452 2319 hybrid_clock.cc:177] HybridClock initialized. Resolution in nanos?: 1 Wait times tolerance adjustment: 1.0005 Current error: 598397 I0109 07:20:30.900527 2319 env_posix.cc:1286] Not raising process file limit of 65536; it is already as high as it can go I0109 07:20:30.900574 2319 file_cache.cc:401] Constructed file cache lbm with capacity 32768 I0109 07:20:30.901990 2319 fs_manager.cc:251] Opened local filesystem: /var/lib/kudu/tserver uuid: "c5a3e756029444b2bf66ee0044f5bc35" format_stamp: "Formatted at 2017-01-09 15:20:11 on aserbin-kudu-jepsen-ts-1" I0109 07:20:30.907037 2319 ts_tablet_manager.cc:701] Loading metadata for tablet 9e99b710634a44f4a948132a5a180600 I0109 07:20:30.907462 2319 ts_tablet_manager.cc:863] Registered tablet 9e99b710634a44f4a948132a5a180600 I0109 07:20:30.907665 2333 ts_tablet_manager.cc:725] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35: Bootstrapping tablet I0109 07:20:30.907959 2333 tablet_bootstrap.cc:381] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35: Bootstrap starting. I0109 07:20:30.908320 2333 tablet_bootstrap.cc:540] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35: Time spent opening tablet: real 0.000s user 0.000s sys 0.000s I0109 07:20:30.908454 2333 tablet_bootstrap.cc:596] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35: Will attempt to recover log segment /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600/wal-000000001 to /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery/wal-000000001 I0109 07:20:30.908490 2333 tablet_bootstrap.cc:604] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35: Moving log directory /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600 to recovery directory /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery in preparation for log replay I0109 07:20:30.908812 2333 log_util.cc:315] Log segment /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery/wal-000000001 has no footer. This segment was likely being written when the server previously shut down. I0109 07:20:30.908829 2333 log_reader.cc:160] Log segment /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery/wal-000000001 was likely left in-progress after a previous crash. Will try to rebuild footer by scanning data. I0109 07:20:30.910146 2319 tablet_server_main.cc:66] Starting tablet server... I0109 07:20:30.913708 2333 log_util.cc:569] Scanning /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery/wal-000000001 for valid entry headers following offset 11219... I0109 07:20:31.042532 2333 log_util.cc:606] Found no log entry headers I0109 07:20:31.042645 2333 log_util.cc:218] Ignoring log segment corruption in /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery/wal-000000001 because there are no log entries following the corrupted one. The server probably crashed in the middle of writing an entry to the write-ahead log or downloaded an active log via tablet copy. Error detail: Corruption: CRC mismatch in log entry header: Log file corruption detected. Failed trying to read batch #0 at offset 11207 for log segment /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery/wal-000000001: Prior entries: [off=10942 REPLICATE (1.48)] [off=10976 COMMIT (1.48)] [off=11173 REPLICATE (1.49)] [off=11207 COMMIT (1.49)] I0109 07:20:31.042713 2333 log_util.cc:368] Successfully rebuilt footer for segment: /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery/wal-000000001 (valid entries through byte offset 11207) I0109 07:20:31.042817 2333 tablet.cc:983] T 9e99b710634a44f4a948132a5a180600 Rewinding schema during bootstrap to Schema [ 0:key[string NOT NULL], 1:value[int32 NOT NULL] ] I0109 07:20:31.043167 2333 log.cc:351] Log is configured to *not* fsync() on all Append() calls I0109 07:20:31.049048 2333 tablet_bootstrap.cc:381] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35: Bootstrap replayed 1/1 log segments. Stats: ops{read=49 overwritten=0 applied=49 ignored=0} inserts{seen=48 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I0109 07:20:31.049126 2333 tablet_bootstrap.cc:1017] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35: ReplayState: Previous OpId: 1.49, Committed OpId: 1.49, Pending Replicates: 0, Pending Commits: 0 I0109 07:20:31.050045 2333 tablet_bootstrap.cc:633] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35: Preparing to delete log recovery files and directory /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery I0109 07:20:31.050065 2333 tablet_bootstrap.cc:636] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35: Renaming log recovery dir from /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery to /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery-1483975231050064 I0109 07:20:31.050117 2333 tablet_bootstrap.cc:646] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35: Deleting all files from renamed log recovery directory /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery-1483975231050064 I0109 07:20:31.050333 2333 tablet_bootstrap.cc:649] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35: Completed deletion of old log recovery files and directory /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery-1483975231050064 I0109 07:20:31.050369 2333 tablet_bootstrap.cc:381] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35: Bootstrap complete. I0109 07:20:31.059664 2333 ts_tablet_manager.cc:730] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35: Time spent bootstrapping tablet: real 0.152s user 0.024s sys 0.036s I0109 07:20:31.060626 2333 raft_consensus.cc:287] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "710c236995db46699eb9055c19ac506e" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-4.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "1678276a36f849c28459de6b6788965a" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-5.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "591b1a1d383d4b659d7549653f4f136e" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-2.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "c5a3e756029444b2bf66ee0044f5bc35" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-1.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "7897578c54854ba0a5af32eb2aa04704" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-3.gce.cloudera.com" port: 7050 } } I0109 07:20:31.060669 2333 raft_consensus.cc:547] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5a3e756029444b2bf66ee0044f5bc35, State: 1, Role: FOLLOWER I0109 07:20:31.060693 2333 consensus_queue.cc:175] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 49, Last appended: 1.49, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER I0109 07:20:31.060952 2333 ts_tablet_manager.cc:757] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s I0109 07:20:31.068023 2319 rpc_server.cc:164] RPC server started. Bound to: 0.0.0.0:7050 I0109 07:20:31.068148 2319 webserver.cc:126] Starting webserver on 0.0.0.0:8050 I0109 07:20:31.068169 2319 webserver.cc:137] Document root disabled I0109 07:20:31.068392 2319 webserver.cc:221] Webserver started. Bound to: http://0.0.0.0:8050/ I0109 07:20:31.068995 2319 tablet_server_main.cc:69] Tablet server successfully started. I0109 07:20:31.112594 2406 raft_consensus.cc:1691] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 591b1a1d383d4b659d7549653f4f136e for term 2 because replica has last-logged OpId of term: 1 index: 49, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 46. I0109 07:20:31.121434 2449 heartbeater.cc:292] Connected to a master server at aserbin-kudu-jepsen-master.gce.cloudera.com:7051 I0109 07:20:31.121558 2449 heartbeater.cc:359] Registering TS with master... I0109 07:20:31.121700 2449 heartbeater.cc:373] Master aserbin-kudu-jepsen-master.gce.cloudera.com:7051 requested a full tablet report, sending... I0109 07:20:31.208112 2406 raft_consensus.cc:1691] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7897578c54854ba0a5af32eb2aa04704 for term 2 because replica has last-logged OpId of term: 1 index: 49, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 48. I0109 07:20:31.277520 2406 raft_consensus.cc:2158] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 0.190s I0109 07:20:31.277598 2406 raft_consensus.cc:1744] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 710c236995db46699eb9055c19ac506e in term 1. I0109 07:20:31.477021 2406 raft_consensus.cc:2202] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35 [term 1 FOLLOWER]: Advancing to term 2 I0109 07:20:31.477156 2406 raft_consensus.cc:2158] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35 [term 2 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 0.182s I0109 07:20:31.477713 2406 raft_consensus.cc:1744] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 710c236995db46699eb9055c19ac506e in term 2. I0109 07:20:31.873945 2386 raft_consensus.cc:410] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0109 07:20:31.874052 2386 raft_consensus.cc:2158] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35 [term 2 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 0.221s I0109 07:20:31.874145 2386 raft_consensus.cc:434] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "710c236995db46699eb9055c19ac506e" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-4.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "1678276a36f849c28459de6b6788965a" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-5.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "591b1a1d383d4b659d7549653f4f136e" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-2.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "c5a3e756029444b2bf66ee0044f5bc35" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-1.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "7897578c54854ba0a5af32eb2aa04704" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-3.gce.cloudera.com" port: 7050 } } I0109 07:20:32.071292 2405 raft_consensus.cc:1719] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1678276a36f849c28459de6b6788965a for term 3 because replica is already servicing an update from a current leader or another vote. I0109 07:20:32.071499 2406 raft_consensus.cc:1691] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 591b1a1d383d4b659d7549653f4f136e for term 3 because replica has last-logged OpId of term: 1 index: 49, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 46. I0109 07:20:32.071660 2386 leader_election.cc:215] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35 [CANDIDATE]: Term 3 pre-election: Requesting pre-vote from peer 710c236995db46699eb9055c19ac506e I0109 07:20:32.071918 2386 leader_election.cc:215] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35 [CANDIDATE]: Term 3 pre-election: Requesting pre-vote from peer 1678276a36f849c28459de6b6788965a I0109 07:20:32.072178 2386 leader_election.cc:215] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35 [CANDIDATE]: Term 3 pre-election: Requesting pre-vote from peer 591b1a1d383d4b659d7549653f4f136e I0109 07:20:32.072284 2386 leader_election.cc:215] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35 [CANDIDATE]: Term 3 pre-election: Requesting pre-vote from peer 7897578c54854ba0a5af32eb2aa04704 I0109 07:20:32.074081 2329 leader_election.cc:347] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35 [CANDIDATE]: Term 3 pre-election: Vote granted by peer 1678276a36f849c28459de6b6788965a I0109 07:20:32.074561 2331 leader_election.cc:347] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35 [CANDIDATE]: Term 3 pre-election: Vote granted by peer 591b1a1d383d4b659d7549653f4f136e I0109 07:20:32.074589 2331 leader_election.cc:242] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. I0109 07:20:32.074729 2463 raft_consensus.cc:2158] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35 [term 2 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 0.222s I0109 07:20:32.074748 2463 raft_consensus.cc:1968] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35 [term 2 FOLLOWER]: Leader pre-election won for term 3 I0109 07:20:32.074755 2463 raft_consensus.cc:410] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0109 07:20:32.074765 2463 raft_consensus.cc:2158] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35 [term 2 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 0.164s I0109 07:20:32.074770 2463 raft_consensus.cc:2202] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35 [term 2 FOLLOWER]: Advancing to term 3 I0109 07:20:32.075196 2463 raft_consensus.cc:434] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "710c236995db46699eb9055c19ac506e" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-4.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "1678276a36f849c28459de6b6788965a" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-5.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "591b1a1d383d4b659d7549653f4f136e" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-2.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "c5a3e756029444b2bf66ee0044f5bc35" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-1.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "7897578c54854ba0a5af32eb2aa04704" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-3.gce.cloudera.com" port: 7050 } } I0109 07:20:32.122831 2449 heartbeater.cc:365] Master aserbin-kudu-jepsen-master.gce.cloudera.com:7051 was elected leader, sending a full tablet report... W0109 07:20:32.293277 2330 leader_election.cc:271] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 710c236995db46699eb9055c19ac506e: Timed out: RequestConsensusVote RPC to 172.31.114.38:7050 timed out after 0.221s (SENT) W0109 07:20:32.294318 2331 leader_election.cc:271] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7897578c54854ba0a5af32eb2aa04704: Timed out: RequestConsensusVote RPC to 172.31.114.37:7050 timed out after 0.221s (SENT) W0109 07:20:32.588627 2463 net_util.cc:124] Time spent resolving address for aserbin-kudu-jepsen-ts-2.gce.cloudera.com: real 0.406s user 0.000s sys 0.000s I0109 07:20:32.637928 2405 raft_consensus.cc:1674] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7897578c54854ba0a5af32eb2aa04704 in current term 3: Already voted for candidate c5a3e756029444b2bf66ee0044f5bc35 in this term. W0109 07:20:32.638077 2405 rpcz_store.cc:234] Call kudu.consensus.ConsensusService.RequestConsensusVote from 172.31.114.37:41028 (request call id 2) took 455ms (client timeout 303). W0109 07:20:32.638145 2405 rpcz_store.cc:238] Trace: 0109 07:20:32.182762 (+ 0us) service_pool.cc:143] Inserting onto call queue 0109 07:20:32.182794 (+ 32us) service_pool.cc:202] Handling call 0109 07:20:32.638046 (+455252us) spinlock_profiling.cc:255] Waited 455 ms on lock 0x449f7b0. stack: 0000000001a20b21 00000000009d8b77 00000000007e48af 0000000000a8971f 0000000000a8a342 0000000001a2dc19 00007f7d9fbc00a3 00007f7d9e45087c ffffffffffffffff 0109 07:20:32.638061 (+ 15us) inbound_call.cc:130] Queueing success response Metrics: {"spinlock_wait_cycles":1137659648} I0109 07:20:32.638183 2386 raft_consensus.cc:410] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0109 07:20:32.638207 2386 raft_consensus.cc:2158] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35 [term 3 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 0.196s I0109 07:20:32.638284 2386 raft_consensus.cc:434] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "710c236995db46699eb9055c19ac506e" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-4.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "1678276a36f849c28459de6b6788965a" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-5.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "591b1a1d383d4b659d7549653f4f136e" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-2.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "c5a3e756029444b2bf66ee0044f5bc35" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-1.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "7897578c54854ba0a5af32eb2aa04704" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-3.gce.cloudera.com" port: 7050 } } I0109 07:20:32.638499 2463 leader_election.cc:215] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35 [CANDIDATE]: Term 3 election: Requesting vote from peer 710c236995db46699eb9055c19ac506e I0109 07:20:32.638592 2463 leader_election.cc:215] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35 [CANDIDATE]: Term 3 election: Requesting vote from peer 1678276a36f849c28459de6b6788965a I0109 07:20:32.638664 2463 leader_election.cc:215] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35 [CANDIDATE]: Term 3 election: Requesting vote from peer 591b1a1d383d4b659d7549653f4f136e I0109 07:20:32.638759 2463 leader_election.cc:215] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35 [CANDIDATE]: Term 3 election: Requesting vote from peer 7897578c54854ba0a5af32eb2aa04704 I0109 07:20:32.650751 2331 leader_election.cc:347] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35 [CANDIDATE]: Term 3 election: Vote granted by peer 7897578c54854ba0a5af32eb2aa04704 W0109 07:20:32.802749 2330 leader_election.cc:271] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 710c236995db46699eb9055c19ac506e: Timed out: RequestConsensusVote RPC to 172.31.114.38:7050 timed out after 0.164s (SENT) W0109 07:20:32.803143 2329 leader_election.cc:271] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 1678276a36f849c28459de6b6788965a: Timed out: RequestConsensusVote RPC to 172.31.114.39:7050 timed out after 0.164s (SENT) W0109 07:20:32.803177 2331 leader_election.cc:271] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 591b1a1d383d4b659d7549653f4f136e: Timed out: RequestConsensusVote RPC to 172.31.113.2:7050 timed out after 0.164s (SENT) I0109 07:20:32.803187 2331 leader_election.cc:242] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. I0109 07:20:32.835211 2386 leader_election.cc:215] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35 [CANDIDATE]: Term 4 pre-election: Requesting pre-vote from peer 710c236995db46699eb9055c19ac506e I0109 07:20:32.835311 2386 leader_election.cc:215] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35 [CANDIDATE]: Term 4 pre-election: Requesting pre-vote from peer 1678276a36f849c28459de6b6788965a I0109 07:20:32.835340 2386 leader_election.cc:215] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35 [CANDIDATE]: Term 4 pre-election: Requesting pre-vote from peer 591b1a1d383d4b659d7549653f4f136e I0109 07:20:32.835376 2386 leader_election.cc:215] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35 [CANDIDATE]: Term 4 pre-election: Requesting pre-vote from peer 7897578c54854ba0a5af32eb2aa04704 I0109 07:20:32.835597 2404 raft_consensus.cc:1719] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1678276a36f849c28459de6b6788965a for term 3 because replica is already servicing an update from a current leader or another vote. I0109 07:20:32.835741 2405 raft_consensus.cc:1674] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7897578c54854ba0a5af32eb2aa04704 in current term 3: Already voted for candidate c5a3e756029444b2bf66ee0044f5bc35 in this term. I0109 07:20:32.835979 2464 raft_consensus.cc:2158] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35 [term 3 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 0.154s I0109 07:20:32.836002 2464 raft_consensus.cc:1905] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority I0109 07:20:32.836066 2406 raft_consensus.cc:1719] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 591b1a1d383d4b659d7549653f4f136e for term 3 because replica is already servicing an update from a current leader or another vote. I0109 07:20:32.836580 2329 leader_election.cc:347] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35 [CANDIDATE]: Term 4 pre-election: Vote granted by peer 1678276a36f849c28459de6b6788965a I0109 07:20:32.836613 2331 leader_election.cc:347] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35 [CANDIDATE]: Term 4 pre-election: Vote granted by peer 591b1a1d383d4b659d7549653f4f136e I0109 07:20:32.836628 2331 leader_election.cc:242] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. I0109 07:20:32.836655 2331 leader_election.cc:347] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35 [CANDIDATE]: Term 4 pre-election: Vote granted by peer 7897578c54854ba0a5af32eb2aa04704 I0109 07:20:32.836678 2464 raft_consensus.cc:2158] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35 [term 3 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 0.318s I0109 07:20:32.836686 2464 raft_consensus.cc:1968] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35 [term 3 FOLLOWER]: Leader pre-election won for term 4 I0109 07:20:32.836693 2464 raft_consensus.cc:410] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0109 07:20:32.836701 2464 raft_consensus.cc:2158] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35 [term 3 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 0.312s I0109 07:20:32.836706 2464 raft_consensus.cc:2202] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35 [term 3 FOLLOWER]: Advancing to term 4 I0109 07:20:32.837215 2464 raft_consensus.cc:434] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "710c236995db46699eb9055c19ac506e" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-4.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "1678276a36f849c28459de6b6788965a" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-5.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "591b1a1d383d4b659d7549653f4f136e" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-2.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "c5a3e756029444b2bf66ee0044f5bc35" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-1.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "7897578c54854ba0a5af32eb2aa04704" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-3.gce.cloudera.com" port: 7050 } } W0109 07:20:33.032395 2330 leader_election.cc:271] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 710c236995db46699eb9055c19ac506e: Timed out: RequestConsensusVote RPC to 172.31.114.38:7050 timed out after 0.196s (SENT) W0109 07:20:37.941247 2464 net_util.cc:124] Time spent resolving address for aserbin-kudu-jepsen-ts-5.gce.cloudera.com: real 5.055s user 0.000s sys 0.000s I0109 07:20:38.042914 2464 leader_election.cc:215] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35 [CANDIDATE]: Term 4 election: Requesting vote from peer 710c236995db46699eb9055c19ac506e I0109 07:20:38.043002 2464 leader_election.cc:215] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35 [CANDIDATE]: Term 4 election: Requesting vote from peer 1678276a36f849c28459de6b6788965a I0109 07:20:38.043052 2464 leader_election.cc:215] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35 [CANDIDATE]: Term 4 election: Requesting vote from peer 591b1a1d383d4b659d7549653f4f136e I0109 07:20:38.043071 2464 leader_election.cc:215] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35 [CANDIDATE]: Term 4 election: Requesting vote from peer 7897578c54854ba0a5af32eb2aa04704 I0109 07:20:38.043114 2386 raft_consensus.cc:410] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0109 07:20:38.043143 2386 raft_consensus.cc:2158] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35 [term 4 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 0.262s I0109 07:20:38.043215 2386 raft_consensus.cc:434] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "710c236995db46699eb9055c19ac506e" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-4.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "1678276a36f849c28459de6b6788965a" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-5.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "591b1a1d383d4b659d7549653f4f136e" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-2.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "c5a3e756029444b2bf66ee0044f5bc35" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-1.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "7897578c54854ba0a5af32eb2aa04704" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-3.gce.cloudera.com" port: 7050 } } W0109 07:20:38.044258 2329 leader_election.cc:271] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 1678276a36f849c28459de6b6788965a: Network error: Client connection negotiation failed: client connection to 172.31.114.39:7050: connect: Connection refused (error 111) W0109 07:20:38.044309 2330 leader_election.cc:271] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 710c236995db46699eb9055c19ac506e: Network error: Client connection negotiation failed: client connection to 172.31.114.38:7050: connect: Connection refused (error 111) W0109 07:20:38.044348 2331 leader_election.cc:271] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 591b1a1d383d4b659d7549653f4f136e: Network error: Client connection negotiation failed: client connection to 172.31.113.2:7050: connect: Connection refused (error 111) I0109 07:20:38.044363 2331 leader_election.cc:242] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. W0109 07:20:38.044404 2331 leader_election.cc:271] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 7897578c54854ba0a5af32eb2aa04704: Network error: Client connection negotiation failed: client connection to 172.31.114.37:7050: connect: Connection refused (error 111) I0109 07:20:43.882355 2482 tablet_server_main.cc:56] Tablet server non-default flags: --raft_heartbeat_interval_ms=50 --fs_data_dirs=/var/lib/kudu/tserver --fs_wal_dir=/var/lib/kudu/tserver --tserver_master_addrs=aserbin-kudu-jepsen-master.gce.cloudera.com:7051 --heap_profile_path=/tmp/kudu-tserver.2482 --unlock_experimental_flags=true --unlock_unsafe_flags=true --flagfile=/etc/kudu/conf/tserver.gflagfile --log_dir=/var/log/kudu --logtostderr=true Tablet server version: kudu 1.3.0-SNAPSHOT revision 3a23b058af0f4de04f8827255de405fff7bf9ee1 build type RELEASE built by jenkins at 09 Jan 2017 06:07:18 PST on aserbin-kudu-jepsen-control-node-debian8.gce.cloudera.com build id 2017-01-09_06-07-06 I0109 07:20:43.882529 2482 mem_tracker.cc:140] MemTracker: hard memory limit is 0.464404 GB I0109 07:20:43.882539 2482 mem_tracker.cc:142] MemTracker: soft memory limit is 0.278643 GB I0109 07:20:43.883693 2482 tablet_server_main.cc:63] Initializing tablet server... I0109 07:20:43.933512 2482 hybrid_clock.cc:177] HybridClock initialized. Resolution in nanos?: 1 Wait times tolerance adjustment: 1.0005 Current error: 604897 I0109 07:20:43.934303 2482 env_posix.cc:1286] Not raising process file limit of 65536; it is already as high as it can go I0109 07:20:43.934332 2482 file_cache.cc:401] Constructed file cache lbm with capacity 32768 I0109 07:20:43.935523 2482 fs_manager.cc:251] Opened local filesystem: /var/lib/kudu/tserver uuid: "c5a3e756029444b2bf66ee0044f5bc35" format_stamp: "Formatted at 2017-01-09 15:20:11 on aserbin-kudu-jepsen-ts-1" I0109 07:20:43.938982 2482 ts_tablet_manager.cc:701] Loading metadata for tablet 9e99b710634a44f4a948132a5a180600 I0109 07:20:43.939214 2482 ts_tablet_manager.cc:863] Registered tablet 9e99b710634a44f4a948132a5a180600 I0109 07:20:43.939400 2496 ts_tablet_manager.cc:725] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35: Bootstrapping tablet I0109 07:20:43.939585 2496 tablet_bootstrap.cc:381] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35: Bootstrap starting. I0109 07:20:43.939874 2496 tablet_bootstrap.cc:540] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35: Time spent opening tablet: real 0.000s user 0.000s sys 0.000s I0109 07:20:43.939975 2496 tablet_bootstrap.cc:596] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35: Will attempt to recover log segment /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600/wal-000000001 to /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery/wal-000000001 I0109 07:20:43.940012 2496 tablet_bootstrap.cc:604] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35: Moving log directory /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600 to recovery directory /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery in preparation for log replay I0109 07:20:43.940227 2496 log_util.cc:315] Log segment /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery/wal-000000001 has no footer. This segment was likely being written when the server previously shut down. I0109 07:20:43.940249 2496 log_reader.cc:160] Log segment /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery/wal-000000001 was likely left in-progress after a previous crash. Will try to rebuild footer by scanning data. I0109 07:20:43.941694 2482 tablet_server_main.cc:66] Starting tablet server... I0109 07:20:43.943176 2496 log_util.cc:569] Scanning /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery/wal-000000001 for valid entry headers following offset 11219... I0109 07:20:44.016208 2496 log_util.cc:606] Found no log entry headers I0109 07:20:44.016297 2496 log_util.cc:218] Ignoring log segment corruption in /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery/wal-000000001 because there are no log entries following the corrupted one. The server probably crashed in the middle of writing an entry to the write-ahead log or downloaded an active log via tablet copy. Error detail: Corruption: CRC mismatch in log entry header: Log file corruption detected. Failed trying to read batch #0 at offset 11207 for log segment /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery/wal-000000001: Prior entries: [off=10942 REPLICATE (1.48)] [off=10976 COMMIT (1.48)] [off=11173 REPLICATE (1.49)] [off=11207 COMMIT (1.49)] I0109 07:20:44.016310 2496 log_util.cc:368] Successfully rebuilt footer for segment: /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery/wal-000000001 (valid entries through byte offset 11207) I0109 07:20:44.016376 2496 tablet.cc:983] T 9e99b710634a44f4a948132a5a180600 Rewinding schema during bootstrap to Schema [ 0:key[string NOT NULL], 1:value[int32 NOT NULL] ] I0109 07:20:44.016688 2496 log.cc:351] Log is configured to *not* fsync() on all Append() calls I0109 07:20:44.024020 2496 tablet_bootstrap.cc:381] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35: Bootstrap replayed 1/1 log segments. Stats: ops{read=49 overwritten=0 applied=49 ignored=0} inserts{seen=48 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I0109 07:20:44.024137 2496 tablet_bootstrap.cc:1017] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35: ReplayState: Previous OpId: 1.49, Committed OpId: 1.49, Pending Replicates: 0, Pending Commits: 0 I0109 07:20:44.025226 2496 tablet_bootstrap.cc:633] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35: Preparing to delete log recovery files and directory /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery I0109 07:20:44.025238 2496 tablet_bootstrap.cc:636] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35: Renaming log recovery dir from /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery to /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery-1483975244025238 I0109 07:20:44.025333 2496 tablet_bootstrap.cc:646] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35: Deleting all files from renamed log recovery directory /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery-1483975244025238 I0109 07:20:44.025827 2496 tablet_bootstrap.cc:649] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35: Completed deletion of old log recovery files and directory /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery-1483975244025238 I0109 07:20:44.025845 2496 tablet_bootstrap.cc:381] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35: Bootstrap complete. I0109 07:20:44.031219 2496 ts_tablet_manager.cc:730] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35: Time spent bootstrapping tablet: real 0.092s user 0.012s sys 0.032s I0109 07:20:44.031929 2496 raft_consensus.cc:287] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35 [term 4 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "710c236995db46699eb9055c19ac506e" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-4.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "1678276a36f849c28459de6b6788965a" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-5.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "591b1a1d383d4b659d7549653f4f136e" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-2.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "c5a3e756029444b2bf66ee0044f5bc35" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-1.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "7897578c54854ba0a5af32eb2aa04704" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-3.gce.cloudera.com" port: 7050 } } I0109 07:20:44.031963 2496 raft_consensus.cc:547] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35 [term 4 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5a3e756029444b2bf66ee0044f5bc35, State: 1, Role: FOLLOWER I0109 07:20:44.031976 2496 consensus_queue.cc:175] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 49, Last appended: 1.49, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER I0109 07:20:44.032166 2496 ts_tablet_manager.cc:757] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s I0109 07:20:44.039548 2482 rpc_server.cc:164] RPC server started. Bound to: 0.0.0.0:7050 I0109 07:20:44.039650 2482 webserver.cc:126] Starting webserver on 0.0.0.0:8050 I0109 07:20:44.039659 2482 webserver.cc:137] Document root disabled I0109 07:20:44.039791 2482 webserver.cc:221] Webserver started. Bound to: http://0.0.0.0:8050/ I0109 07:20:44.040184 2482 tablet_server_main.cc:69] Tablet server successfully started. I0109 07:20:44.092954 2612 heartbeater.cc:292] Connected to a master server at aserbin-kudu-jepsen-master.gce.cloudera.com:7051 I0109 07:20:44.093008 2612 heartbeater.cc:359] Registering TS with master... I0109 07:20:44.093104 2612 heartbeater.cc:373] Master aserbin-kudu-jepsen-master.gce.cloudera.com:7051 requested a full tablet report, sending... I0109 07:20:44.182286 2569 raft_consensus.cc:2158] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35 [term 4 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 0.184s I0109 07:20:44.182365 2569 raft_consensus.cc:1744] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 591b1a1d383d4b659d7549653f4f136e in term 4. I0109 07:20:44.539068 2535 raft_consensus.cc:410] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0109 07:20:44.539185 2535 raft_consensus.cc:2158] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35 [term 4 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 0.201s I0109 07:20:44.539290 2535 raft_consensus.cc:434] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "710c236995db46699eb9055c19ac506e" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-4.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "1678276a36f849c28459de6b6788965a" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-5.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "591b1a1d383d4b659d7549653f4f136e" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-2.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "c5a3e756029444b2bf66ee0044f5bc35" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-1.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "7897578c54854ba0a5af32eb2aa04704" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-3.gce.cloudera.com" port: 7050 } } I0109 07:20:44.735024 2535 leader_election.cc:215] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35 [CANDIDATE]: Term 5 pre-election: Requesting pre-vote from peer 710c236995db46699eb9055c19ac506e I0109 07:20:44.735106 2535 leader_election.cc:215] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35 [CANDIDATE]: Term 5 pre-election: Requesting pre-vote from peer 1678276a36f849c28459de6b6788965a I0109 07:20:44.735124 2535 leader_election.cc:215] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35 [CANDIDATE]: Term 5 pre-election: Requesting pre-vote from peer 591b1a1d383d4b659d7549653f4f136e I0109 07:20:44.735136 2535 leader_election.cc:215] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35 [CANDIDATE]: Term 5 pre-election: Requesting pre-vote from peer 7897578c54854ba0a5af32eb2aa04704 I0109 07:20:44.735524 2569 raft_consensus.cc:2158] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35 [term 4 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 0.193s I0109 07:20:44.735550 2569 raft_consensus.cc:1744] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 710c236995db46699eb9055c19ac506e in term 4. I0109 07:20:44.735798 2568 raft_consensus.cc:1719] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 591b1a1d383d4b659d7549653f4f136e for term 5 because replica is already servicing an update from a current leader or another vote. I0109 07:20:44.737717 2494 leader_election.cc:361] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35 [CANDIDATE]: Term 5 pre-election: Vote denied by peer 7897578c54854ba0a5af32eb2aa04704. Message: Invalid argument: T 9e99b710634a44f4a948132a5a180600 P 7897578c54854ba0a5af32eb2aa04704 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c5a3e756029444b2bf66ee0044f5bc35 for term 5 because replica has last-logged OpId of term: 4 index: 61, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 49. I0109 07:20:44.738082 2492 leader_election.cc:361] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35 [CANDIDATE]: Term 5 pre-election: Vote denied by peer 1678276a36f849c28459de6b6788965a. Message: Invalid argument: T 9e99b710634a44f4a948132a5a180600 P 1678276a36f849c28459de6b6788965a [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c5a3e756029444b2bf66ee0044f5bc35 for term 5 because replica has last-logged OpId of term: 4 index: 61, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 49. I0109 07:20:44.903772 2568 raft_consensus.cc:2202] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35 [term 4 FOLLOWER]: Advancing to term 5 I0109 07:20:44.903877 2568 raft_consensus.cc:2158] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35 [term 5 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 0.190s I0109 07:20:44.904314 2568 raft_consensus.cc:1744] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 710c236995db46699eb9055c19ac506e in term 5. I0109 07:20:44.904391 2494 leader_election.cc:361] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35 [CANDIDATE]: Term 5 pre-election: Vote denied by peer 591b1a1d383d4b659d7549653f4f136e. Message: Invalid argument: T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c5a3e756029444b2bf66ee0044f5bc35 in current term 5: Already voted for candidate 591b1a1d383d4b659d7549653f4f136e in this term. I0109 07:20:44.904404 2494 leader_election.cc:242] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. I0109 07:20:44.904492 2493 leader_election.cc:361] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35 [CANDIDATE]: Term 5 pre-election: Vote denied by peer 710c236995db46699eb9055c19ac506e. Message: Service unavailable: T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c5a3e756029444b2bf66ee0044f5bc35 for term 5 because replica is already servicing an update from a current leader or another vote. I0109 07:20:44.904577 2568 raft_consensus.cc:1674] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 591b1a1d383d4b659d7549653f4f136e in current term 5: Already voted for candidate 710c236995db46699eb9055c19ac506e in this term. I0109 07:20:44.904775 2626 raft_consensus.cc:2158] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35 [term 5 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 0.155s I0109 07:20:44.904790 2626 raft_consensus.cc:1905] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority I0109 07:20:45.094195 2612 heartbeater.cc:365] Master aserbin-kudu-jepsen-master.gce.cloudera.com:7051 was elected leader, sending a full tablet report... I0109 07:20:45.103845 2568 raft_consensus.cc:915] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35 [term 5 FOLLOWER]: Refusing update from remote peer 710c236995db46699eb9055c19ac506e: Log matching property violated. Preceding OpId in replica: term: 1 index: 49. Preceding OpId from leader: term: 5 index: 62. (index mismatch) I0109 07:21:06.866103 2661 tablet_server_main.cc:56] Tablet server non-default flags: --raft_heartbeat_interval_ms=50 --fs_data_dirs=/var/lib/kudu/tserver --fs_wal_dir=/var/lib/kudu/tserver --tserver_master_addrs=aserbin-kudu-jepsen-master.gce.cloudera.com:7051 --heap_profile_path=/tmp/kudu-tserver.2661 --unlock_experimental_flags=true --unlock_unsafe_flags=true --flagfile=/etc/kudu/conf/tserver.gflagfile --log_dir=/var/log/kudu --logtostderr=true Tablet server version: kudu 1.3.0-SNAPSHOT revision 3a23b058af0f4de04f8827255de405fff7bf9ee1 build type RELEASE built by jenkins at 09 Jan 2017 06:07:18 PST on aserbin-kudu-jepsen-control-node-debian8.gce.cloudera.com build id 2017-01-09_06-07-06 I0109 07:21:06.866271 2661 mem_tracker.cc:140] MemTracker: hard memory limit is 0.464404 GB I0109 07:21:06.866282 2661 mem_tracker.cc:142] MemTracker: soft memory limit is 0.278643 GB I0109 07:21:06.867414 2661 tablet_server_main.cc:63] Initializing tablet server... I0109 07:21:06.916750 2661 hybrid_clock.cc:177] HybridClock initialized. Resolution in nanos?: 1 Wait times tolerance adjustment: 1.0005 Current error: 616397 I0109 07:21:06.917551 2661 env_posix.cc:1286] Not raising process file limit of 65536; it is already as high as it can go I0109 07:21:06.917583 2661 file_cache.cc:401] Constructed file cache lbm with capacity 32768 I0109 07:21:06.918728 2661 fs_manager.cc:251] Opened local filesystem: /var/lib/kudu/tserver uuid: "c5a3e756029444b2bf66ee0044f5bc35" format_stamp: "Formatted at 2017-01-09 15:20:11 on aserbin-kudu-jepsen-ts-1" I0109 07:21:06.921937 2661 ts_tablet_manager.cc:701] Loading metadata for tablet 9e99b710634a44f4a948132a5a180600 I0109 07:21:06.922168 2661 ts_tablet_manager.cc:863] Registered tablet 9e99b710634a44f4a948132a5a180600 I0109 07:21:06.922428 2675 ts_tablet_manager.cc:725] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35: Bootstrapping tablet I0109 07:21:06.922662 2675 tablet_bootstrap.cc:381] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35: Bootstrap starting. I0109 07:21:06.922920 2675 tablet_bootstrap.cc:540] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35: Time spent opening tablet: real 0.000s user 0.000s sys 0.000s I0109 07:21:06.923001 2675 tablet_bootstrap.cc:596] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35: Will attempt to recover log segment /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600/wal-000000001 to /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery/wal-000000001 I0109 07:21:06.923025 2675 tablet_bootstrap.cc:604] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35: Moving log directory /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600 to recovery directory /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery in preparation for log replay I0109 07:21:06.923221 2675 log_util.cc:315] Log segment /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery/wal-000000001 has no footer. This segment was likely being written when the server previously shut down. I0109 07:21:06.923233 2675 log_reader.cc:160] Log segment /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery/wal-000000001 was likely left in-progress after a previous crash. Will try to rebuild footer by scanning data. I0109 07:21:06.924720 2661 tablet_server_main.cc:66] Starting tablet server... I0109 07:21:06.926704 2675 log_util.cc:569] Scanning /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery/wal-000000001 for valid entry headers following offset 29225... I0109 07:21:06.999168 2675 log_util.cc:606] Found no log entry headers I0109 07:21:06.999254 2675 log_util.cc:218] Ignoring log segment corruption in /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery/wal-000000001 because there are no log entries following the corrupted one. The server probably crashed in the middle of writing an entry to the write-ahead log or downloaded an active log via tablet copy. Error detail: Corruption: CRC mismatch in log entry header: Log file corruption detected. Failed trying to read batch #0 at offset 29213 for log segment /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery/wal-000000001: Prior entries: [off=28945 REPLICATE (5.128)] [off=28980 COMMIT (5.128)] [off=29178 REPLICATE (5.129)] [off=29213 COMMIT (5.129)] I0109 07:21:06.999341 2675 log_util.cc:368] Successfully rebuilt footer for segment: /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery/wal-000000001 (valid entries through byte offset 29213) I0109 07:21:06.999408 2675 tablet.cc:983] T 9e99b710634a44f4a948132a5a180600 Rewinding schema during bootstrap to Schema [ 0:key[string NOT NULL], 1:value[int32 NOT NULL] ] I0109 07:21:06.999691 2675 log.cc:351] Log is configured to *not* fsync() on all Append() calls I0109 07:21:07.008409 2675 tablet_bootstrap.cc:381] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35: Bootstrap replayed 1/1 log segments. Stats: ops{read=129 overwritten=0 applied=129 ignored=0} inserts{seen=126 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I0109 07:21:07.008441 2675 tablet_bootstrap.cc:1017] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35: ReplayState: Previous OpId: 5.129, Committed OpId: 5.129, Pending Replicates: 0, Pending Commits: 0 I0109 07:21:07.008867 2675 tablet_bootstrap.cc:633] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35: Preparing to delete log recovery files and directory /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery I0109 07:21:07.008890 2675 tablet_bootstrap.cc:636] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35: Renaming log recovery dir from /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery to /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery-1483975267008889 I0109 07:21:07.008909 2675 tablet_bootstrap.cc:646] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35: Deleting all files from renamed log recovery directory /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery-1483975267008889 I0109 07:21:07.009029 2675 tablet_bootstrap.cc:649] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35: Completed deletion of old log recovery files and directory /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery-1483975267008889 I0109 07:21:07.009045 2675 tablet_bootstrap.cc:381] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35: Bootstrap complete. I0109 07:21:07.014395 2675 ts_tablet_manager.cc:730] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35: Time spent bootstrapping tablet: real 0.092s user 0.012s sys 0.036s I0109 07:21:07.015087 2675 raft_consensus.cc:287] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35 [term 5 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "710c236995db46699eb9055c19ac506e" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-4.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "1678276a36f849c28459de6b6788965a" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-5.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "591b1a1d383d4b659d7549653f4f136e" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-2.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "c5a3e756029444b2bf66ee0044f5bc35" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-1.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "7897578c54854ba0a5af32eb2aa04704" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-3.gce.cloudera.com" port: 7050 } } I0109 07:21:07.015126 2675 raft_consensus.cc:547] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35 [term 5 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5a3e756029444b2bf66ee0044f5bc35, State: 1, Role: FOLLOWER I0109 07:21:07.015139 2675 consensus_queue.cc:175] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 129, Last appended: 5.129, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER I0109 07:21:07.015342 2675 ts_tablet_manager.cc:757] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s I0109 07:21:07.023130 2661 rpc_server.cc:164] RPC server started. Bound to: 0.0.0.0:7050 I0109 07:21:07.023219 2661 webserver.cc:126] Starting webserver on 0.0.0.0:8050 I0109 07:21:07.023227 2661 webserver.cc:137] Document root disabled I0109 07:21:07.023377 2661 webserver.cc:221] Webserver started. Bound to: http://0.0.0.0:8050/ I0109 07:21:07.023808 2661 tablet_server_main.cc:69] Tablet server successfully started. I0109 07:21:07.076357 2791 heartbeater.cc:292] Connected to a master server at aserbin-kudu-jepsen-master.gce.cloudera.com:7051 I0109 07:21:07.076416 2791 heartbeater.cc:359] Registering TS with master... I0109 07:21:07.076514 2791 heartbeater.cc:373] Master aserbin-kudu-jepsen-master.gce.cloudera.com:7051 requested a full tablet report, sending... I0109 07:21:07.176527 2748 raft_consensus.cc:2158] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35 [term 5 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 0.195s I0109 07:21:07.176580 2748 raft_consensus.cc:1744] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 710c236995db46699eb9055c19ac506e in term 5. I0109 07:21:07.374343 2748 raft_consensus.cc:2202] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35 [term 5 FOLLOWER]: Advancing to term 6 I0109 07:21:07.374435 2748 raft_consensus.cc:2158] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35 [term 6 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 0.183s I0109 07:21:07.374819 2748 raft_consensus.cc:1744] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 710c236995db46699eb9055c19ac506e in term 6. I0109 07:21:07.573109 2748 raft_consensus.cc:915] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35 [term 6 FOLLOWER]: Refusing update from remote peer 710c236995db46699eb9055c19ac506e: Log matching property violated. Preceding OpId in replica: term: 5 index: 129. Preceding OpId from leader: term: 6 index: 162. (index mismatch) I0109 07:21:07.574246 2791 heartbeater.cc:365] Master aserbin-kudu-jepsen-master.gce.cloudera.com:7051 was elected leader, sending a full tablet report... I0109 07:21:25.225739 2819 tablet_server_main.cc:56] Tablet server non-default flags: --raft_heartbeat_interval_ms=50 --fs_data_dirs=/var/lib/kudu/tserver --fs_wal_dir=/var/lib/kudu/tserver --tserver_master_addrs=aserbin-kudu-jepsen-master.gce.cloudera.com:7051 --heap_profile_path=/tmp/kudu-tserver.2819 --unlock_experimental_flags=true --unlock_unsafe_flags=true --flagfile=/etc/kudu/conf/tserver.gflagfile --log_dir=/var/log/kudu --logtostderr=true Tablet server version: kudu 1.3.0-SNAPSHOT revision 3a23b058af0f4de04f8827255de405fff7bf9ee1 build type RELEASE built by jenkins at 09 Jan 2017 06:07:18 PST on aserbin-kudu-jepsen-control-node-debian8.gce.cloudera.com build id 2017-01-09_06-07-06 I0109 07:21:25.225898 2819 mem_tracker.cc:140] MemTracker: hard memory limit is 0.464404 GB I0109 07:21:25.225908 2819 mem_tracker.cc:142] MemTracker: soft memory limit is 0.278643 GB I0109 07:21:25.227180 2819 tablet_server_main.cc:63] Initializing tablet server... I0109 07:21:25.276805 2819 hybrid_clock.cc:177] HybridClock initialized. Resolution in nanos?: 1 Wait times tolerance adjustment: 1.0005 Current error: 625897 I0109 07:21:25.277580 2819 env_posix.cc:1286] Not raising process file limit of 65536; it is already as high as it can go I0109 07:21:25.277609 2819 file_cache.cc:401] Constructed file cache lbm with capacity 32768 I0109 07:21:25.278693 2819 fs_manager.cc:251] Opened local filesystem: /var/lib/kudu/tserver uuid: "c5a3e756029444b2bf66ee0044f5bc35" format_stamp: "Formatted at 2017-01-09 15:20:11 on aserbin-kudu-jepsen-ts-1" I0109 07:21:25.282881 2819 ts_tablet_manager.cc:701] Loading metadata for tablet 9e99b710634a44f4a948132a5a180600 I0109 07:21:25.283128 2819 ts_tablet_manager.cc:863] Registered tablet 9e99b710634a44f4a948132a5a180600 I0109 07:21:25.283299 2833 ts_tablet_manager.cc:725] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35: Bootstrapping tablet I0109 07:21:25.283540 2833 tablet_bootstrap.cc:381] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35: Bootstrap starting. I0109 07:21:25.283792 2833 tablet_bootstrap.cc:540] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35: Time spent opening tablet: real 0.000s user 0.000s sys 0.000s I0109 07:21:25.283870 2833 tablet_bootstrap.cc:596] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35: Will attempt to recover log segment /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600/wal-000000001 to /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery/wal-000000001 I0109 07:21:25.283882 2833 tablet_bootstrap.cc:604] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35: Moving log directory /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600 to recovery directory /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery in preparation for log replay I0109 07:21:25.284062 2833 log_util.cc:315] Log segment /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery/wal-000000001 has no footer. This segment was likely being written when the server previously shut down. I0109 07:21:25.284071 2833 log_reader.cc:160] Log segment /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery/wal-000000001 was likely left in-progress after a previous crash. Will try to rebuild footer by scanning data. I0109 07:21:25.285580 2819 tablet_server_main.cc:66] Starting tablet server... I0109 07:21:25.287827 2833 log_util.cc:569] Scanning /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery/wal-000000001 for valid entry headers following offset 47458... I0109 07:21:25.354020 2833 log_util.cc:606] Found no log entry headers I0109 07:21:25.354109 2833 log_util.cc:218] Ignoring log segment corruption in /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery/wal-000000001 because there are no log entries following the corrupted one. The server probably crashed in the middle of writing an entry to the write-ahead log or downloaded an active log via tablet copy. Error detail: Corruption: CRC mismatch in log entry header: Log file corruption detected. Failed trying to read batch #0 at offset 47446 for log segment /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery/wal-000000001: Prior entries: [off=47178 REPLICATE (6.208)] [off=47213 COMMIT (6.208)] [off=47411 REPLICATE (6.209)] [off=47446 COMMIT (6.209)] I0109 07:21:25.354145 2833 log_util.cc:368] Successfully rebuilt footer for segment: /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery/wal-000000001 (valid entries through byte offset 47446) I0109 07:21:25.354212 2833 tablet.cc:983] T 9e99b710634a44f4a948132a5a180600 Rewinding schema during bootstrap to Schema [ 0:key[string NOT NULL], 1:value[int32 NOT NULL] ] I0109 07:21:25.354493 2833 log.cc:351] Log is configured to *not* fsync() on all Append() calls I0109 07:21:25.369309 2833 tablet_bootstrap.cc:381] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35: Bootstrap replayed 1/1 log segments. Stats: ops{read=209 overwritten=0 applied=209 ignored=0} inserts{seen=205 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I0109 07:21:25.369375 2833 tablet_bootstrap.cc:1017] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35: ReplayState: Previous OpId: 6.209, Committed OpId: 6.209, Pending Replicates: 0, Pending Commits: 0 I0109 07:21:25.369915 2833 tablet_bootstrap.cc:633] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35: Preparing to delete log recovery files and directory /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery I0109 07:21:25.369927 2833 tablet_bootstrap.cc:636] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35: Renaming log recovery dir from /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery to /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery-1483975285369927 I0109 07:21:25.369948 2833 tablet_bootstrap.cc:646] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35: Deleting all files from renamed log recovery directory /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery-1483975285369927 I0109 07:21:25.370101 2833 tablet_bootstrap.cc:649] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35: Completed deletion of old log recovery files and directory /var/lib/kudu/tserver/wals/9e99b710634a44f4a948132a5a180600.recovery-1483975285369927 I0109 07:21:25.370115 2833 tablet_bootstrap.cc:381] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35: Bootstrap complete. I0109 07:21:25.375010 2833 ts_tablet_manager.cc:730] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35: Time spent bootstrapping tablet: real 0.092s user 0.016s sys 0.036s I0109 07:21:25.375516 2833 raft_consensus.cc:287] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35 [term 6 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "710c236995db46699eb9055c19ac506e" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-4.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "1678276a36f849c28459de6b6788965a" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-5.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "591b1a1d383d4b659d7549653f4f136e" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-2.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "c5a3e756029444b2bf66ee0044f5bc35" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-1.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "7897578c54854ba0a5af32eb2aa04704" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-3.gce.cloudera.com" port: 7050 } } I0109 07:21:25.375542 2833 raft_consensus.cc:547] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35 [term 6 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5a3e756029444b2bf66ee0044f5bc35, State: 1, Role: FOLLOWER I0109 07:21:25.375555 2833 consensus_queue.cc:175] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 209, Last appended: 6.209, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER I0109 07:21:25.375802 2833 ts_tablet_manager.cc:757] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s I0109 07:21:25.383522 2819 rpc_server.cc:164] RPC server started. Bound to: 0.0.0.0:7050 I0109 07:21:25.383610 2819 webserver.cc:126] Starting webserver on 0.0.0.0:8050 I0109 07:21:25.383617 2819 webserver.cc:137] Document root disabled I0109 07:21:25.383756 2819 webserver.cc:221] Webserver started. Bound to: http://0.0.0.0:8050/ I0109 07:21:25.384219 2819 tablet_server_main.cc:69] Tablet server successfully started. I0109 07:21:25.436466 2949 heartbeater.cc:292] Connected to a master server at aserbin-kudu-jepsen-master.gce.cloudera.com:7051 I0109 07:21:25.436517 2949 heartbeater.cc:359] Registering TS with master... I0109 07:21:25.436580 2949 heartbeater.cc:373] Master aserbin-kudu-jepsen-master.gce.cloudera.com:7051 requested a full tablet report, sending... I0109 07:21:25.611090 2875 raft_consensus.cc:410] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0109 07:21:25.611198 2875 raft_consensus.cc:2158] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35 [term 6 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 0.150s I0109 07:21:25.611276 2875 raft_consensus.cc:434] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "710c236995db46699eb9055c19ac506e" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-4.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "1678276a36f849c28459de6b6788965a" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-5.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "591b1a1d383d4b659d7549653f4f136e" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-2.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "c5a3e756029444b2bf66ee0044f5bc35" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-1.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "7897578c54854ba0a5af32eb2aa04704" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-3.gce.cloudera.com" port: 7050 } } I0109 07:21:25.807898 2875 leader_election.cc:215] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35 [CANDIDATE]: Term 7 pre-election: Requesting pre-vote from peer 710c236995db46699eb9055c19ac506e I0109 07:21:25.808231 2875 leader_election.cc:215] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35 [CANDIDATE]: Term 7 pre-election: Requesting pre-vote from peer 1678276a36f849c28459de6b6788965a I0109 07:21:25.808408 2875 leader_election.cc:215] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35 [CANDIDATE]: Term 7 pre-election: Requesting pre-vote from peer 591b1a1d383d4b659d7549653f4f136e I0109 07:21:25.808562 2875 leader_election.cc:215] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35 [CANDIDATE]: Term 7 pre-election: Requesting pre-vote from peer 7897578c54854ba0a5af32eb2aa04704 W0109 07:21:25.809420 2830 leader_election.cc:271] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 710c236995db46699eb9055c19ac506e: Network error: Client connection negotiation failed: client connection to 172.31.114.38:7050: connect: Connection refused (error 111) I0109 07:21:25.810559 2829 leader_election.cc:347] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35 [CANDIDATE]: Term 7 pre-election: Vote granted by peer 1678276a36f849c28459de6b6788965a I0109 07:21:25.810801 2831 leader_election.cc:347] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35 [CANDIDATE]: Term 7 pre-election: Vote granted by peer 7897578c54854ba0a5af32eb2aa04704 I0109 07:21:25.810816 2831 leader_election.cc:242] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. I0109 07:21:25.810974 2955 raft_consensus.cc:2158] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35 [term 6 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 0.202s I0109 07:21:25.811002 2955 raft_consensus.cc:1968] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35 [term 6 FOLLOWER]: Leader pre-election won for term 7 I0109 07:21:25.811028 2955 raft_consensus.cc:410] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0109 07:21:25.811039 2955 raft_consensus.cc:2158] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35 [term 6 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 0.165s I0109 07:21:25.811045 2955 raft_consensus.cc:2202] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35 [term 6 FOLLOWER]: Advancing to term 7 I0109 07:21:25.811449 2955 raft_consensus.cc:434] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "710c236995db46699eb9055c19ac506e" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-4.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "1678276a36f849c28459de6b6788965a" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-5.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "591b1a1d383d4b659d7549653f4f136e" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-2.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "c5a3e756029444b2bf66ee0044f5bc35" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-1.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "7897578c54854ba0a5af32eb2aa04704" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-3.gce.cloudera.com" port: 7050 } } I0109 07:21:25.942687 2831 leader_election.cc:361] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35 [CANDIDATE]: Term 7 pre-election: Vote denied by peer 591b1a1d383d4b659d7549653f4f136e. Message: Invalid argument: T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c5a3e756029444b2bf66ee0044f5bc35 for term 7 because replica has last-logged OpId of term: 6 index: 210, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 209. I0109 07:21:26.008929 2955 leader_election.cc:215] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35 [CANDIDATE]: Term 7 election: Requesting vote from peer 710c236995db46699eb9055c19ac506e I0109 07:21:26.009027 2955 leader_election.cc:215] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35 [CANDIDATE]: Term 7 election: Requesting vote from peer 1678276a36f849c28459de6b6788965a I0109 07:21:26.009055 2955 leader_election.cc:215] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35 [CANDIDATE]: Term 7 election: Requesting vote from peer 591b1a1d383d4b659d7549653f4f136e I0109 07:21:26.009086 2955 leader_election.cc:215] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35 [CANDIDATE]: Term 7 election: Requesting vote from peer 7897578c54854ba0a5af32eb2aa04704 I0109 07:21:26.009683 2906 raft_consensus.cc:1674] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 591b1a1d383d4b659d7549653f4f136e in current term 7: Already voted for candidate c5a3e756029444b2bf66ee0044f5bc35 in this term. I0109 07:21:26.010499 2829 leader_election.cc:347] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35 [CANDIDATE]: Term 7 election: Vote granted by peer 1678276a36f849c28459de6b6788965a I0109 07:21:26.010761 2831 leader_election.cc:347] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35 [CANDIDATE]: Term 7 election: Vote granted by peer 7897578c54854ba0a5af32eb2aa04704 I0109 07:21:26.010777 2831 leader_election.cc:242] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. I0109 07:21:26.010848 2955 raft_consensus.cc:2158] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35 [term 7 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 0.201s I0109 07:21:26.010896 2955 raft_consensus.cc:1968] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35 [term 7 FOLLOWER]: Leader election won for term 7 I0109 07:21:26.010931 2955 raft_consensus.cc:517] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35 [term 7 LEADER]: Becoming Leader. State: Replica: c5a3e756029444b2bf66ee0044f5bc35, State: 1, Role: LEADER I0109 07:21:26.011013 2955 consensus_queue.cc:157] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 209, Committed index: 209, Last appended: 6.209, Current term: 7, Majority size: 3, State: 1, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "710c236995db46699eb9055c19ac506e" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-4.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "1678276a36f849c28459de6b6788965a" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-5.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "591b1a1d383d4b659d7549653f4f136e" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-2.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "c5a3e756029444b2bf66ee0044f5bc35" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-1.gce.cloudera.com" port: 7050 } } peers { permanent_uuid: "7897578c54854ba0a5af32eb2aa04704" member_type: VOTER last_known_addr { host: "aserbin-kudu-jepsen-ts-3.gce.cloudera.com" port: 7050 } } I0109 07:21:26.011615 2830 leader_election.cc:361] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35 [CANDIDATE]: Term 7 election: Vote denied by peer 710c236995db46699eb9055c19ac506e. Message: Invalid argument: T 9e99b710634a44f4a948132a5a180600 P 710c236995db46699eb9055c19ac506e [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate c5a3e756029444b2bf66ee0044f5bc35 for term 7 because replica has last-logged OpId of term: 6 index: 210, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 209. I0109 07:21:26.142905 2831 leader_election.cc:361] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35 [CANDIDATE]: Term 7 election: Vote denied by peer 591b1a1d383d4b659d7549653f4f136e. Message: Invalid argument: T 9e99b710634a44f4a948132a5a180600 P 591b1a1d383d4b659d7549653f4f136e [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate c5a3e756029444b2bf66ee0044f5bc35 in current term 7: Already voted for candidate 591b1a1d383d4b659d7549653f4f136e in this term. I0109 07:21:26.209652 2906 raft_consensus.cc:1705] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35 [term 7 LEADER]: Leader election vote request: Denying vote to candidate 591b1a1d383d4b659d7549653f4f136e for term 7 because replica is either leader or believes a valid leader to be alive. I0109 07:21:26.209797 2949 heartbeater.cc:365] Master aserbin-kudu-jepsen-master.gce.cloudera.com:7051 was elected leader, sending a full tablet report... I0109 07:21:26.210664 2955 consensus_queue.cc:693] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35 [LEADER]: Connected to new peer: Peer: 710c236995db46699eb9055c19ac506e, Is new: false, Last received: 6.209, Next index: 210, Last known committed idx: 209, Last exchange result: ERROR, Needs tablet copy: false I0109 07:21:26.210814 2955 consensus_queue.cc:693] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35 [LEADER]: Connected to new peer: Peer: 1678276a36f849c28459de6b6788965a, Is new: false, Last received: 6.209, Next index: 210, Last known committed idx: 209, Last exchange result: ERROR, Needs tablet copy: false I0109 07:21:26.210841 2955 consensus_queue.cc:693] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35 [LEADER]: Connected to new peer: Peer: 591b1a1d383d4b659d7549653f4f136e, Is new: false, Last received: 6.209, Next index: 210, Last known committed idx: 209, Last exchange result: ERROR, Needs tablet copy: false I0109 07:21:26.212527 2955 consensus_queue.cc:693] T 9e99b710634a44f4a948132a5a180600 P c5a3e756029444b2bf66ee0044f5bc35 [LEADER]: Connected to new peer: Peer: 7897578c54854ba0a5af32eb2aa04704, Is new: false, Last received: 6.206, Next index: 207, Last known committed idx: 206, Last exchange result: ERROR, Needs tablet copy: false