I1031 23:03:52.449784 5853 tablet_service.cc:650] Processing CreateTablet for tablet fcde65c4e4cf4df29b9ef9884ce292b2 (table=default.timeout_tst [id=cdcb3e60c1314776843d856365535d49]), partition=HASH (id) PARTITION 2, RANGE (date) PARTITION 20171031 <= VALUES < 20171101 I1031 23:03:52.449822 5870 tablet_service.cc:650] Processing CreateTablet for tablet 865b2d5788f44598b169ca6415314610 (table=default.timeout_tst [id=cdcb3e60c1314776843d856365535d49]), partition=HASH (id) PARTITION 3, RANGE (date) PARTITION 20171031 <= VALUES < 20171101 I1031 23:03:52.449854 5866 tablet_service.cc:650] Processing CreateTablet for tablet e003f6e546374517aa9a51b4bc23c827 (table=default.timeout_tst [id=cdcb3e60c1314776843d856365535d49]), partition=HASH (id) PARTITION 3, RANGE (date) PARTITION 20171024 <= VALUES < 20171025 I1031 23:03:52.449862 5869 tablet_service.cc:650] Processing CreateTablet for tablet de45d05c3a064dc5acb91d66dcaea6ed (table=default.timeout_tst [id=cdcb3e60c1314776843d856365535d49]), partition=HASH (id) PARTITION 3, RANGE (date) PARTITION 20171029 <= VALUES < 20171030 I1031 23:03:52.450116 5861 tablet_service.cc:650] Processing CreateTablet for tablet 6a813ca355f749b6b31a773aa0f2ac7a (table=default.timeout_tst [id=cdcb3e60c1314776843d856365535d49]), partition=HASH (id) PARTITION 4, RANGE (date) PARTITION 20171027 <= VALUES < 20171028 I1031 23:03:52.450265 5871 tablet_service.cc:650] Processing CreateTablet for tablet 648eb376255a4dd4bb5818262efcc813 (table=default.timeout_tst [id=cdcb3e60c1314776843d856365535d49]), partition=HASH (id) PARTITION 4, RANGE (date) PARTITION 20171101 <= VALUES < 20171102 I1031 23:03:52.450304 5857 tablet_service.cc:650] Processing CreateTablet for tablet 42d56cd12bbb4bed9973f96a230fdb9e (table=default.timeout_tst [id=cdcb3e60c1314776843d856365535d49]), partition=HASH (id) PARTITION 4, RANGE (date) PARTITION 20171030 <= VALUES < 20171031 I1031 23:03:52.771405 5871 ts_tablet_manager.cc:862] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd: Registered tablet (data state: TABLET_DATA_READY) I1031 23:03:52.771951 34047 ts_tablet_manager.cc:723] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd: Bootstrapping tablet I1031 23:03:52.772532 34047 tablet_bootstrap.cc:382] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd: Bootstrap starting. I1031 23:03:52.772704 34047 tablet_bootstrap.cc:541] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd: Time spent opening tablet: real 0.000s user 0.000s sys 0.000s I1031 23:03:52.772848 34047 tablet_bootstrap.cc:484] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd: No blocks or log segments found. Creating new log. I1031 23:03:52.785292 34047 tablet_bootstrap.cc:382] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd: No bootstrap required, opened a new log I1031 23:03:52.785328 34047 ts_tablet_manager.cc:728] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd: Time spent bootstrapping tablet: real 0.013s user 0.001s sys 0.000s I1031 23:03:52.786170 34047 raft_consensus.cc:288] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3eb30bd1bc34b95902fa61724239553" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st05.bj" port: 18700 } } peers { permanent_uuid: "0a8bbcbbb5a6459aa5258464755db3bb" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st02.bj" port: 18700 } } peers { permanent_uuid: "0f53a0d3ef7e44ebb0365c800752d5bd" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st01.bj" port: 18700 } } I1031 23:03:52.786195 34047 raft_consensus.cc:319] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1031 23:03:52.786206 34047 raft_consensus.cc:548] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f53a0d3ef7e44ebb0365c800752d5bd, State: 1, Role: FOLLOWER I1031 23:03:52.786224 34047 consensus_queue.cc:176] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER I1031 23:03:52.813303 5870 ts_tablet_manager.cc:862] T 865b2d5788f44598b169ca6415314610 P 0f53a0d3ef7e44ebb0365c800752d5bd: Registered tablet (data state: TABLET_DATA_READY) I1031 23:03:52.813469 34052 ts_tablet_manager.cc:723] T 865b2d5788f44598b169ca6415314610 P 0f53a0d3ef7e44ebb0365c800752d5bd: Bootstrapping tablet I1031 23:03:52.813964 34052 tablet_bootstrap.cc:382] T 865b2d5788f44598b169ca6415314610 P 0f53a0d3ef7e44ebb0365c800752d5bd: Bootstrap starting. I1031 23:03:52.814108 34052 tablet_bootstrap.cc:541] T 865b2d5788f44598b169ca6415314610 P 0f53a0d3ef7e44ebb0365c800752d5bd: Time spent opening tablet: real 0.000s user 0.000s sys 0.000s I1031 23:03:52.814216 34052 tablet_bootstrap.cc:484] T 865b2d5788f44598b169ca6415314610 P 0f53a0d3ef7e44ebb0365c800752d5bd: No blocks or log segments found. Creating new log. I1031 23:03:52.815469 34052 tablet_bootstrap.cc:382] T 865b2d5788f44598b169ca6415314610 P 0f53a0d3ef7e44ebb0365c800752d5bd: No bootstrap required, opened a new log I1031 23:03:52.815495 34052 ts_tablet_manager.cc:728] T 865b2d5788f44598b169ca6415314610 P 0f53a0d3ef7e44ebb0365c800752d5bd: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s I1031 23:03:52.816205 34052 raft_consensus.cc:288] T 865b2d5788f44598b169ca6415314610 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f53a0d3ef7e44ebb0365c800752d5bd" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st01.bj" port: 18700 } } peers { permanent_uuid: "daba39963d4346e1816089fd3a5442b5" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st09.bj" port: 18700 } } peers { permanent_uuid: "0a8bbcbbb5a6459aa5258464755db3bb" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st02.bj" port: 18700 } } I1031 23:03:52.816227 34052 raft_consensus.cc:319] T 865b2d5788f44598b169ca6415314610 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1031 23:03:52.816236 34052 raft_consensus.cc:548] T 865b2d5788f44598b169ca6415314610 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f53a0d3ef7e44ebb0365c800752d5bd, State: 1, Role: FOLLOWER I1031 23:03:52.816251 34052 consensus_queue.cc:176] T 865b2d5788f44598b169ca6415314610 P 0f53a0d3ef7e44ebb0365c800752d5bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER I1031 23:03:52.838371 5861 ts_tablet_manager.cc:862] T 6a813ca355f749b6b31a773aa0f2ac7a P 0f53a0d3ef7e44ebb0365c800752d5bd: Registered tablet (data state: TABLET_DATA_READY) I1031 23:03:52.838902 34062 ts_tablet_manager.cc:723] T 6a813ca355f749b6b31a773aa0f2ac7a P 0f53a0d3ef7e44ebb0365c800752d5bd: Bootstrapping tablet I1031 23:03:52.839473 34062 tablet_bootstrap.cc:382] T 6a813ca355f749b6b31a773aa0f2ac7a P 0f53a0d3ef7e44ebb0365c800752d5bd: Bootstrap starting. I1031 23:03:52.839622 34062 tablet_bootstrap.cc:541] T 6a813ca355f749b6b31a773aa0f2ac7a P 0f53a0d3ef7e44ebb0365c800752d5bd: Time spent opening tablet: real 0.000s user 0.000s sys 0.000s I1031 23:03:52.839751 34062 tablet_bootstrap.cc:484] T 6a813ca355f749b6b31a773aa0f2ac7a P 0f53a0d3ef7e44ebb0365c800752d5bd: No blocks or log segments found. Creating new log. I1031 23:03:52.841331 34062 tablet_bootstrap.cc:382] T 6a813ca355f749b6b31a773aa0f2ac7a P 0f53a0d3ef7e44ebb0365c800752d5bd: No bootstrap required, opened a new log I1031 23:03:52.841367 34062 ts_tablet_manager.cc:728] T 6a813ca355f749b6b31a773aa0f2ac7a P 0f53a0d3ef7e44ebb0365c800752d5bd: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s I1031 23:03:52.842074 34062 raft_consensus.cc:288] T 6a813ca355f749b6b31a773aa0f2ac7a P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a8bbcbbb5a6459aa5258464755db3bb" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st02.bj" port: 18700 } } peers { permanent_uuid: "0f53a0d3ef7e44ebb0365c800752d5bd" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st01.bj" port: 18700 } } peers { permanent_uuid: "193181be29284287b25df2ee92938f0b" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st08.bj" port: 18700 } } I1031 23:03:52.842097 34062 raft_consensus.cc:319] T 6a813ca355f749b6b31a773aa0f2ac7a P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1031 23:03:52.842105 34062 raft_consensus.cc:548] T 6a813ca355f749b6b31a773aa0f2ac7a P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f53a0d3ef7e44ebb0365c800752d5bd, State: 1, Role: FOLLOWER I1031 23:03:52.842118 34062 consensus_queue.cc:176] T 6a813ca355f749b6b31a773aa0f2ac7a P 0f53a0d3ef7e44ebb0365c800752d5bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER I1031 23:03:52.863608 5866 ts_tablet_manager.cc:862] T e003f6e546374517aa9a51b4bc23c827 P 0f53a0d3ef7e44ebb0365c800752d5bd: Registered tablet (data state: TABLET_DATA_READY) I1031 23:03:52.864140 34087 ts_tablet_manager.cc:723] T e003f6e546374517aa9a51b4bc23c827 P 0f53a0d3ef7e44ebb0365c800752d5bd: Bootstrapping tablet I1031 23:03:52.864531 34087 tablet_bootstrap.cc:382] T e003f6e546374517aa9a51b4bc23c827 P 0f53a0d3ef7e44ebb0365c800752d5bd: Bootstrap starting. I1031 23:03:52.864644 34087 tablet_bootstrap.cc:541] T e003f6e546374517aa9a51b4bc23c827 P 0f53a0d3ef7e44ebb0365c800752d5bd: Time spent opening tablet: real 0.000s user 0.000s sys 0.000s I1031 23:03:52.864733 34087 tablet_bootstrap.cc:484] T e003f6e546374517aa9a51b4bc23c827 P 0f53a0d3ef7e44ebb0365c800752d5bd: No blocks or log segments found. Creating new log. I1031 23:03:52.865855 34087 tablet_bootstrap.cc:382] T e003f6e546374517aa9a51b4bc23c827 P 0f53a0d3ef7e44ebb0365c800752d5bd: No bootstrap required, opened a new log I1031 23:03:52.865881 34087 ts_tablet_manager.cc:728] T e003f6e546374517aa9a51b4bc23c827 P 0f53a0d3ef7e44ebb0365c800752d5bd: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s I1031 23:03:52.866325 34087 raft_consensus.cc:288] T e003f6e546374517aa9a51b4bc23c827 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f53a0d3ef7e44ebb0365c800752d5bd" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st01.bj" port: 18700 } } peers { permanent_uuid: "0fc477287c154136abc1e2f3c82fb7f9" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st04.bj" port: 18700 } } peers { permanent_uuid: "044f1da9a27c46acb82b1386f829f4dc" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st03.bj" port: 18700 } } I1031 23:03:52.866344 34087 raft_consensus.cc:319] T e003f6e546374517aa9a51b4bc23c827 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1031 23:03:52.866355 34087 raft_consensus.cc:548] T e003f6e546374517aa9a51b4bc23c827 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f53a0d3ef7e44ebb0365c800752d5bd, State: 1, Role: FOLLOWER I1031 23:03:52.866370 34087 consensus_queue.cc:176] T e003f6e546374517aa9a51b4bc23c827 P 0f53a0d3ef7e44ebb0365c800752d5bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER I1031 23:03:52.889009 5857 ts_tablet_manager.cc:862] T 42d56cd12bbb4bed9973f96a230fdb9e P 0f53a0d3ef7e44ebb0365c800752d5bd: Registered tablet (data state: TABLET_DATA_READY) I1031 23:03:52.889497 34121 ts_tablet_manager.cc:723] T 42d56cd12bbb4bed9973f96a230fdb9e P 0f53a0d3ef7e44ebb0365c800752d5bd: Bootstrapping tablet I1031 23:03:52.889912 34121 tablet_bootstrap.cc:382] T 42d56cd12bbb4bed9973f96a230fdb9e P 0f53a0d3ef7e44ebb0365c800752d5bd: Bootstrap starting. I1031 23:03:52.890038 34121 tablet_bootstrap.cc:541] T 42d56cd12bbb4bed9973f96a230fdb9e P 0f53a0d3ef7e44ebb0365c800752d5bd: Time spent opening tablet: real 0.000s user 0.000s sys 0.000s I1031 23:03:52.890139 34121 tablet_bootstrap.cc:484] T 42d56cd12bbb4bed9973f96a230fdb9e P 0f53a0d3ef7e44ebb0365c800752d5bd: No blocks or log segments found. Creating new log. I1031 23:03:52.891207 34121 tablet_bootstrap.cc:382] T 42d56cd12bbb4bed9973f96a230fdb9e P 0f53a0d3ef7e44ebb0365c800752d5bd: No bootstrap required, opened a new log I1031 23:03:52.891235 34121 ts_tablet_manager.cc:728] T 42d56cd12bbb4bed9973f96a230fdb9e P 0f53a0d3ef7e44ebb0365c800752d5bd: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s I1031 23:03:52.891664 34121 raft_consensus.cc:288] T 42d56cd12bbb4bed9973f96a230fdb9e P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3eb30bd1bc34b95902fa61724239553" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st05.bj" port: 18700 } } peers { permanent_uuid: "0f53a0d3ef7e44ebb0365c800752d5bd" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st01.bj" port: 18700 } } peers { permanent_uuid: "23f962e4a1744381ad5fa0d2d8b10241" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st07.bj" port: 18700 } } I1031 23:03:52.891685 34121 raft_consensus.cc:319] T 42d56cd12bbb4bed9973f96a230fdb9e P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1031 23:03:52.891698 34121 raft_consensus.cc:548] T 42d56cd12bbb4bed9973f96a230fdb9e P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f53a0d3ef7e44ebb0365c800752d5bd, State: 1, Role: FOLLOWER I1031 23:03:52.891717 34121 consensus_queue.cc:176] T 42d56cd12bbb4bed9973f96a230fdb9e P 0f53a0d3ef7e44ebb0365c800752d5bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER I1031 23:03:52.914000 5869 ts_tablet_manager.cc:862] T de45d05c3a064dc5acb91d66dcaea6ed P 0f53a0d3ef7e44ebb0365c800752d5bd: Registered tablet (data state: TABLET_DATA_READY) I1031 23:03:52.914528 34126 ts_tablet_manager.cc:723] T de45d05c3a064dc5acb91d66dcaea6ed P 0f53a0d3ef7e44ebb0365c800752d5bd: Bootstrapping tablet I1031 23:03:52.914968 34126 tablet_bootstrap.cc:382] T de45d05c3a064dc5acb91d66dcaea6ed P 0f53a0d3ef7e44ebb0365c800752d5bd: Bootstrap starting. I1031 23:03:52.915099 34126 tablet_bootstrap.cc:541] T de45d05c3a064dc5acb91d66dcaea6ed P 0f53a0d3ef7e44ebb0365c800752d5bd: Time spent opening tablet: real 0.000s user 0.000s sys 0.000s I1031 23:03:52.915195 34126 tablet_bootstrap.cc:484] T de45d05c3a064dc5acb91d66dcaea6ed P 0f53a0d3ef7e44ebb0365c800752d5bd: No blocks or log segments found. Creating new log. I1031 23:03:52.916019 34126 tablet_bootstrap.cc:382] T de45d05c3a064dc5acb91d66dcaea6ed P 0f53a0d3ef7e44ebb0365c800752d5bd: No bootstrap required, opened a new log I1031 23:03:52.916045 34126 ts_tablet_manager.cc:728] T de45d05c3a064dc5acb91d66dcaea6ed P 0f53a0d3ef7e44ebb0365c800752d5bd: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s I1031 23:03:52.916795 34126 raft_consensus.cc:288] T de45d05c3a064dc5acb91d66dcaea6ed P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "193181be29284287b25df2ee92938f0b" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st08.bj" port: 18700 } } peers { permanent_uuid: "0f53a0d3ef7e44ebb0365c800752d5bd" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st01.bj" port: 18700 } } peers { permanent_uuid: "23f962e4a1744381ad5fa0d2d8b10241" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st07.bj" port: 18700 } } I1031 23:03:52.916815 34126 raft_consensus.cc:319] T de45d05c3a064dc5acb91d66dcaea6ed P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1031 23:03:52.916823 34126 raft_consensus.cc:548] T de45d05c3a064dc5acb91d66dcaea6ed P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f53a0d3ef7e44ebb0365c800752d5bd, State: 1, Role: FOLLOWER I1031 23:03:52.916836 34126 consensus_queue.cc:176] T de45d05c3a064dc5acb91d66dcaea6ed P 0f53a0d3ef7e44ebb0365c800752d5bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER I1031 23:03:52.947528 5853 ts_tablet_manager.cc:862] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd: Registered tablet (data state: TABLET_DATA_READY) I1031 23:03:52.948041 34134 ts_tablet_manager.cc:723] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd: Bootstrapping tablet I1031 23:03:52.948405 34134 tablet_bootstrap.cc:382] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd: Bootstrap starting. I1031 23:03:52.948519 34134 tablet_bootstrap.cc:541] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd: Time spent opening tablet: real 0.000s user 0.000s sys 0.000s I1031 23:03:52.948611 34134 tablet_bootstrap.cc:484] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd: No blocks or log segments found. Creating new log. I1031 23:03:52.949084 34134 tablet_bootstrap.cc:382] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd: No bootstrap required, opened a new log I1031 23:03:52.949110 34134 ts_tablet_manager.cc:728] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s I1031 23:03:52.949569 34134 raft_consensus.cc:288] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a8bbcbbb5a6459aa5258464755db3bb" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st02.bj" port: 18700 } } peers { permanent_uuid: "0f53a0d3ef7e44ebb0365c800752d5bd" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st01.bj" port: 18700 } } peers { permanent_uuid: "23f962e4a1744381ad5fa0d2d8b10241" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st07.bj" port: 18700 } } I1031 23:03:52.949590 34134 raft_consensus.cc:319] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1031 23:03:52.949601 34134 raft_consensus.cc:548] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f53a0d3ef7e44ebb0365c800752d5bd, State: 1, Role: FOLLOWER I1031 23:03:52.949628 34134 consensus_queue.cc:176] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER I1031 23:03:53.054920 5881 raft_consensus.cc:2162] T de45d05c3a064dc5acb91d66dcaea6ed P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.713s I1031 23:03:53.054998 5881 raft_consensus.cc:1747] T de45d05c3a064dc5acb91d66dcaea6ed P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23f962e4a1744381ad5fa0d2d8b10241 in term 0. I1031 23:03:53.056243 5881 raft_consensus.cc:2206] T de45d05c3a064dc5acb91d66dcaea6ed P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Advancing to term 1 I1031 23:03:53.056309 5881 raft_consensus.cc:2162] T de45d05c3a064dc5acb91d66dcaea6ed P 0f53a0d3ef7e44ebb0365c800752d5bd [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 2.140s I1031 23:03:53.056919 5881 raft_consensus.cc:1747] T de45d05c3a064dc5acb91d66dcaea6ed P 0f53a0d3ef7e44ebb0365c800752d5bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23f962e4a1744381ad5fa0d2d8b10241 in term 1. I1031 23:03:53.070168 34050 raft_consensus.cc:411] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I1031 23:03:53.070212 34050 raft_consensus.cc:2162] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.668s I1031 23:03:53.070258 34050 raft_consensus.cc:435] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3eb30bd1bc34b95902fa61724239553" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st05.bj" port: 18700 } } peers { permanent_uuid: "0a8bbcbbb5a6459aa5258464755db3bb" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st02.bj" port: 18700 } } peers { permanent_uuid: "0f53a0d3ef7e44ebb0365c800752d5bd" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st01.bj" port: 18700 } } I1031 23:03:53.071054 34050 leader_election.cc:216] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer d3eb30bd1bc34b95902fa61724239553 I1031 23:03:53.071104 34050 leader_election.cc:216] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 0a8bbcbbb5a6459aa5258464755db3bb W1031 23:03:53.071379 5816 leader_election.cc:278] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0a8bbcbbb5a6459aa5258464755db3bb: Not found: Tablet not found I1031 23:03:53.071499 5815 leader_election.cc:348] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Vote granted by peer d3eb30bd1bc34b95902fa61724239553 I1031 23:03:53.071538 5815 leader_election.cc:243] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. I1031 23:03:53.071612 34051 raft_consensus.cc:2162] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.947s I1031 23:03:53.071635 34051 raft_consensus.cc:1971] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Leader pre-election won for term 1 I1031 23:03:53.071652 34051 raft_consensus.cc:411] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I1031 23:03:53.071666 34051 raft_consensus.cc:2162] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.848s I1031 23:03:53.071673 34051 raft_consensus.cc:2206] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Advancing to term 1 I1031 23:03:53.072160 34051 raft_consensus.cc:435] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3eb30bd1bc34b95902fa61724239553" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st05.bj" port: 18700 } } peers { permanent_uuid: "0a8bbcbbb5a6459aa5258464755db3bb" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st02.bj" port: 18700 } } peers { permanent_uuid: "0f53a0d3ef7e44ebb0365c800752d5bd" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st01.bj" port: 18700 } } I1031 23:03:53.072816 34051 leader_election.cc:216] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 election: Requesting vote from peer d3eb30bd1bc34b95902fa61724239553 I1031 23:03:53.072868 34051 leader_election.cc:216] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 election: Requesting vote from peer 0a8bbcbbb5a6459aa5258464755db3bb W1031 23:03:53.073072 5816 leader_election.cc:278] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0a8bbcbbb5a6459aa5258464755db3bb: Not found: Tablet not found I1031 23:03:53.073853 5815 leader_election.cc:348] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 election: Vote granted by peer d3eb30bd1bc34b95902fa61724239553 I1031 23:03:53.073873 5815 leader_election.cc:243] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I1031 23:03:53.073933 34051 raft_consensus.cc:2162] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.575s I1031 23:03:53.073956 34051 raft_consensus.cc:1971] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 1 FOLLOWER]: Leader election won for term 1 I1031 23:03:53.074599 34051 raft_consensus.cc:518] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 1 LEADER]: Becoming Leader. State: Replica: 0f53a0d3ef7e44ebb0365c800752d5bd, State: 1, Role: LEADER I1031 23:03:53.074671 34051 consensus_queue.cc:158] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Current term: 1, Majority size: 2, State: 1, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3eb30bd1bc34b95902fa61724239553" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st05.bj" port: 18700 } } peers { permanent_uuid: "0a8bbcbbb5a6459aa5258464755db3bb" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st02.bj" port: 18700 } } peers { permanent_uuid: "0f53a0d3ef7e44ebb0365c800752d5bd" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st01.bj" port: 18700 } } I1031 23:03:53.129585 34137 raft_consensus.cc:411] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I1031 23:03:53.129617 34137 raft_consensus.cc:2162] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 2.017s I1031 23:03:53.129660 34137 raft_consensus.cc:435] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a8bbcbbb5a6459aa5258464755db3bb" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st02.bj" port: 18700 } } peers { permanent_uuid: "0f53a0d3ef7e44ebb0365c800752d5bd" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st01.bj" port: 18700 } } peers { permanent_uuid: "23f962e4a1744381ad5fa0d2d8b10241" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st07.bj" port: 18700 } } I1031 23:03:53.130363 34137 leader_election.cc:216] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 0a8bbcbbb5a6459aa5258464755db3bb I1031 23:03:53.130403 34137 leader_election.cc:216] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 23f962e4a1744381ad5fa0d2d8b10241 W1031 23:03:53.130563 5816 leader_election.cc:278] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 23f962e4a1744381ad5fa0d2d8b10241: Not found: Tablet not found I1031 23:03:53.130667 5816 leader_election.cc:348] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Vote granted by peer 0a8bbcbbb5a6459aa5258464755db3bb I1031 23:03:53.130686 5816 leader_election.cc:243] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. I1031 23:03:53.130725 34138 raft_consensus.cc:2162] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.792s I1031 23:03:53.130743 34138 raft_consensus.cc:1971] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Leader pre-election won for term 1 I1031 23:03:53.130753 34138 raft_consensus.cc:411] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I1031 23:03:53.130764 34138 raft_consensus.cc:2162] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.872s I1031 23:03:53.130774 34138 raft_consensus.cc:2206] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Advancing to term 1 I1031 23:03:53.131276 34138 raft_consensus.cc:435] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a8bbcbbb5a6459aa5258464755db3bb" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st02.bj" port: 18700 } } peers { permanent_uuid: "0f53a0d3ef7e44ebb0365c800752d5bd" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st01.bj" port: 18700 } } peers { permanent_uuid: "23f962e4a1744381ad5fa0d2d8b10241" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st07.bj" port: 18700 } } I1031 23:03:53.131747 34138 leader_election.cc:216] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 election: Requesting vote from peer 0a8bbcbbb5a6459aa5258464755db3bb I1031 23:03:53.131777 34138 leader_election.cc:216] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 election: Requesting vote from peer 23f962e4a1744381ad5fa0d2d8b10241 W1031 23:03:53.131958 5816 leader_election.cc:278] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 23f962e4a1744381ad5fa0d2d8b10241: Not found: Tablet not found I1031 23:03:53.159550 5888 raft_consensus.cc:2162] T e003f6e546374517aa9a51b4bc23c827 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.974s I1031 23:03:53.159615 5888 raft_consensus.cc:1747] T e003f6e546374517aa9a51b4bc23c827 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 044f1da9a27c46acb82b1386f829f4dc in term 0. I1031 23:03:53.160182 5816 leader_election.cc:348] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 election: Vote granted by peer 0a8bbcbbb5a6459aa5258464755db3bb I1031 23:03:53.160202 5816 leader_election.cc:243] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I1031 23:03:53.160249 34138 raft_consensus.cc:2162] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.654s I1031 23:03:53.160270 34138 raft_consensus.cc:1971] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 1 FOLLOWER]: Leader election won for term 1 I1031 23:03:53.160455 34138 raft_consensus.cc:518] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 1 LEADER]: Becoming Leader. State: Replica: 0f53a0d3ef7e44ebb0365c800752d5bd, State: 1, Role: LEADER I1031 23:03:53.160523 34138 consensus_queue.cc:158] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Current term: 1, Majority size: 2, State: 1, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a8bbcbbb5a6459aa5258464755db3bb" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st02.bj" port: 18700 } } peers { permanent_uuid: "0f53a0d3ef7e44ebb0365c800752d5bd" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st01.bj" port: 18700 } } peers { permanent_uuid: "23f962e4a1744381ad5fa0d2d8b10241" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st07.bj" port: 18700 } } I1031 23:03:53.161439 5888 raft_consensus.cc:2206] T e003f6e546374517aa9a51b4bc23c827 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Advancing to term 1 I1031 23:03:53.161468 5888 raft_consensus.cc:2162] T e003f6e546374517aa9a51b4bc23c827 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 2.209s I1031 23:03:53.161967 5888 raft_consensus.cc:1747] T e003f6e546374517aa9a51b4bc23c827 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 044f1da9a27c46acb82b1386f829f4dc in term 1. I1031 23:03:53.167086 34067 raft_consensus.cc:411] T 6a813ca355f749b6b31a773aa0f2ac7a P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I1031 23:03:53.167114 34067 raft_consensus.cc:2162] T 6a813ca355f749b6b31a773aa0f2ac7a P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 2.073s I1031 23:03:53.167160 34067 raft_consensus.cc:435] T 6a813ca355f749b6b31a773aa0f2ac7a P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a8bbcbbb5a6459aa5258464755db3bb" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st02.bj" port: 18700 } } peers { permanent_uuid: "0f53a0d3ef7e44ebb0365c800752d5bd" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st01.bj" port: 18700 } } peers { permanent_uuid: "193181be29284287b25df2ee92938f0b" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st08.bj" port: 18700 } } I1031 23:03:53.167845 34067 leader_election.cc:216] T 6a813ca355f749b6b31a773aa0f2ac7a P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 0a8bbcbbb5a6459aa5258464755db3bb I1031 23:03:53.167878 34067 leader_election.cc:216] T 6a813ca355f749b6b31a773aa0f2ac7a P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 193181be29284287b25df2ee92938f0b W1031 23:03:53.168087 5815 leader_election.cc:278] T 6a813ca355f749b6b31a773aa0f2ac7a P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 193181be29284287b25df2ee92938f0b: Not found: Tablet not found W1031 23:03:53.168117 5816 leader_election.cc:278] T 6a813ca355f749b6b31a773aa0f2ac7a P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0a8bbcbbb5a6459aa5258464755db3bb: Not found: Tablet not found I1031 23:03:53.168138 5816 leader_election.cc:243] T 6a813ca355f749b6b31a773aa0f2ac7a P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. I1031 23:03:53.168185 34068 raft_consensus.cc:2162] T 6a813ca355f749b6b31a773aa0f2ac7a P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.838s I1031 23:03:53.168200 34068 raft_consensus.cc:1908] T 6a813ca355f749b6b31a773aa0f2ac7a P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority I1031 23:03:53.235081 5888 raft_consensus.cc:2162] T 42d56cd12bbb4bed9973f96a230fdb9e P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.878s I1031 23:03:53.235100 5888 raft_consensus.cc:1747] T 42d56cd12bbb4bed9973f96a230fdb9e P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3eb30bd1bc34b95902fa61724239553 in term 0. I1031 23:03:53.236636 5888 raft_consensus.cc:2206] T 42d56cd12bbb4bed9973f96a230fdb9e P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Advancing to term 1 I1031 23:03:53.236661 5888 raft_consensus.cc:2162] T 42d56cd12bbb4bed9973f96a230fdb9e P 0f53a0d3ef7e44ebb0365c800752d5bd [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 2.217s I1031 23:03:53.237105 5888 raft_consensus.cc:1747] T 42d56cd12bbb4bed9973f96a230fdb9e P 0f53a0d3ef7e44ebb0365c800752d5bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3eb30bd1bc34b95902fa61724239553 in term 1. I1031 23:03:53.360508 5888 raft_consensus.cc:2162] T 865b2d5788f44598b169ca6415314610 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 2.033s I1031 23:03:53.360546 5888 raft_consensus.cc:1747] T 865b2d5788f44598b169ca6415314610 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate daba39963d4346e1816089fd3a5442b5 in term 0. I1031 23:03:53.361938 5888 raft_consensus.cc:2206] T 865b2d5788f44598b169ca6415314610 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Advancing to term 1 I1031 23:03:53.361979 5888 raft_consensus.cc:2162] T 865b2d5788f44598b169ca6415314610 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.662s I1031 23:03:53.362407 5888 raft_consensus.cc:1747] T 865b2d5788f44598b169ca6415314610 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate daba39963d4346e1816089fd3a5442b5 in term 1. I1031 23:03:53.558444 5889 raft_consensus.cc:918] T de45d05c3a064dc5acb91d66dcaea6ed P 0f53a0d3ef7e44ebb0365c800752d5bd [term 1 FOLLOWER]: Refusing update from remote peer 23f962e4a1744381ad5fa0d2d8b10241: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch) I1031 23:03:53.576048 34051 consensus_queue.cc:695] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd [LEADER]: Connected to new peer: Peer: d3eb30bd1bc34b95902fa61724239553, Is new: false, Last received: 0.0, Next index: 1, Last known committed idx: 0, Last exchange result: ERROR, Needs tablet copy: false W1031 23:03:53.576369 5816 consensus_peers.cc:357] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 0a8bbcbbb5a6459aa5258464755db3bb (c3-kudu-tst-st02.bj:18700): Couldn't send request to peer 0a8bbcbbb5a6459aa5258464755db3bb for tablet 648eb376255a4dd4bb5818262efcc813. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 1 times. I1031 23:03:53.661762 34138 consensus_queue.cc:695] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd [LEADER]: Connected to new peer: Peer: 0a8bbcbbb5a6459aa5258464755db3bb, Is new: false, Last received: 0.0, Next index: 1, Last known committed idx: 0, Last exchange result: ERROR, Needs tablet copy: false W1031 23:03:53.662385 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 1 times. I1031 23:03:53.663113 5888 raft_consensus.cc:918] T e003f6e546374517aa9a51b4bc23c827 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 1 FOLLOWER]: Refusing update from remote peer 044f1da9a27c46acb82b1386f829f4dc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch) I1031 23:03:53.738533 5888 raft_consensus.cc:918] T 42d56cd12bbb4bed9973f96a230fdb9e P 0f53a0d3ef7e44ebb0365c800752d5bd [term 1 FOLLOWER]: Refusing update from remote peer d3eb30bd1bc34b95902fa61724239553: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch) I1031 23:03:53.863497 5881 raft_consensus.cc:918] T 865b2d5788f44598b169ca6415314610 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 1 FOLLOWER]: Refusing update from remote peer daba39963d4346e1816089fd3a5442b5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch) W1031 23:03:54.076472 5816 consensus_peers.cc:357] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 0a8bbcbbb5a6459aa5258464755db3bb (c3-kudu-tst-st02.bj:18700): Couldn't send request to peer 0a8bbcbbb5a6459aa5258464755db3bb for tablet 648eb376255a4dd4bb5818262efcc813. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 2 times. W1031 23:03:54.162510 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 2 times. W1031 23:03:54.333564 5816 consensus_peers.cc:357] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 0a8bbcbbb5a6459aa5258464755db3bb (c3-kudu-tst-st02.bj:18700): Couldn't send request to peer 0a8bbcbbb5a6459aa5258464755db3bb for tablet 648eb376255a4dd4bb5818262efcc813. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 3 times. W1031 23:03:54.580587 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 3 times. W1031 23:03:54.833601 5816 consensus_peers.cc:357] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 0a8bbcbbb5a6459aa5258464755db3bb (c3-kudu-tst-st02.bj:18700): Couldn't send request to peer 0a8bbcbbb5a6459aa5258464755db3bb for tablet 648eb376255a4dd4bb5818262efcc813. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 4 times. W1031 23:03:55.080744 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 4 times. W1031 23:03:55.333694 5816 consensus_peers.cc:357] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 0a8bbcbbb5a6459aa5258464755db3bb (c3-kudu-tst-st02.bj:18700): Couldn't send request to peer 0a8bbcbbb5a6459aa5258464755db3bb for tablet 648eb376255a4dd4bb5818262efcc813. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 5 times. W1031 23:03:55.580739 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 5 times. W1031 23:03:55.833748 5816 consensus_peers.cc:357] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 0a8bbcbbb5a6459aa5258464755db3bb (c3-kudu-tst-st02.bj:18700): Couldn't send request to peer 0a8bbcbbb5a6459aa5258464755db3bb for tablet 648eb376255a4dd4bb5818262efcc813. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 6 times. W1031 23:03:56.080938 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 6 times. W1031 23:03:56.333851 5816 consensus_peers.cc:357] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 0a8bbcbbb5a6459aa5258464755db3bb (c3-kudu-tst-st02.bj:18700): Couldn't send request to peer 0a8bbcbbb5a6459aa5258464755db3bb for tablet 648eb376255a4dd4bb5818262efcc813. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 7 times. W1031 23:03:56.580988 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 7 times. I1031 23:03:56.707510 34067 raft_consensus.cc:411] T 6a813ca355f749b6b31a773aa0f2ac7a P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I1031 23:03:56.707545 34067 raft_consensus.cc:2162] T 6a813ca355f749b6b31a773aa0f2ac7a P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.639s I1031 23:03:56.707629 34067 raft_consensus.cc:435] T 6a813ca355f749b6b31a773aa0f2ac7a P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a8bbcbbb5a6459aa5258464755db3bb" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st02.bj" port: 18700 } } peers { permanent_uuid: "0f53a0d3ef7e44ebb0365c800752d5bd" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st01.bj" port: 18700 } } peers { permanent_uuid: "193181be29284287b25df2ee92938f0b" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st08.bj" port: 18700 } } I1031 23:03:56.708287 34067 leader_election.cc:216] T 6a813ca355f749b6b31a773aa0f2ac7a P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 0a8bbcbbb5a6459aa5258464755db3bb I1031 23:03:56.708319 34067 leader_election.cc:216] T 6a813ca355f749b6b31a773aa0f2ac7a P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 193181be29284287b25df2ee92938f0b W1031 23:03:56.708642 5816 leader_election.cc:278] T 6a813ca355f749b6b31a773aa0f2ac7a P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0a8bbcbbb5a6459aa5258464755db3bb: Illegal state: Tablet not RUNNING: NOT_STARTED W1031 23:03:56.708680 5815 leader_election.cc:278] T 6a813ca355f749b6b31a773aa0f2ac7a P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 193181be29284287b25df2ee92938f0b: Illegal state: Tablet not RUNNING: NOT_STARTED I1031 23:03:56.708701 5815 leader_election.cc:243] T 6a813ca355f749b6b31a773aa0f2ac7a P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. I1031 23:03:56.708906 34243 raft_consensus.cc:2162] T 6a813ca355f749b6b31a773aa0f2ac7a P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.760s I1031 23:03:56.708923 34243 raft_consensus.cc:1908] T 6a813ca355f749b6b31a773aa0f2ac7a P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority W1031 23:03:56.833983 5816 consensus_peers.cc:357] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 0a8bbcbbb5a6459aa5258464755db3bb (c3-kudu-tst-st02.bj:18700): Couldn't send request to peer 0a8bbcbbb5a6459aa5258464755db3bb for tablet 648eb376255a4dd4bb5818262efcc813. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 8 times. W1031 23:03:57.081097 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 8 times. W1031 23:03:57.334025 5816 consensus_peers.cc:357] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 0a8bbcbbb5a6459aa5258464755db3bb (c3-kudu-tst-st02.bj:18700): Couldn't send request to peer 0a8bbcbbb5a6459aa5258464755db3bb for tablet 648eb376255a4dd4bb5818262efcc813. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 9 times. W1031 23:03:57.581127 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 9 times. W1031 23:03:57.834102 5816 consensus_peers.cc:357] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 0a8bbcbbb5a6459aa5258464755db3bb (c3-kudu-tst-st02.bj:18700): Couldn't send request to peer 0a8bbcbbb5a6459aa5258464755db3bb for tablet 648eb376255a4dd4bb5818262efcc813. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 10 times. W1031 23:03:58.081075 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 10 times. W1031 23:03:58.334182 5816 consensus_peers.cc:357] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 0a8bbcbbb5a6459aa5258464755db3bb (c3-kudu-tst-st02.bj:18700): Couldn't send request to peer 0a8bbcbbb5a6459aa5258464755db3bb for tablet 648eb376255a4dd4bb5818262efcc813. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 11 times. W1031 23:03:58.581197 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 11 times. W1031 23:03:58.834195 5816 consensus_peers.cc:357] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 0a8bbcbbb5a6459aa5258464755db3bb (c3-kudu-tst-st02.bj:18700): Couldn't send request to peer 0a8bbcbbb5a6459aa5258464755db3bb for tablet 648eb376255a4dd4bb5818262efcc813. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 12 times. W1031 23:03:59.081313 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 12 times. W1031 23:03:59.334213 5816 consensus_peers.cc:357] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 0a8bbcbbb5a6459aa5258464755db3bb (c3-kudu-tst-st02.bj:18700): Couldn't send request to peer 0a8bbcbbb5a6459aa5258464755db3bb for tablet 648eb376255a4dd4bb5818262efcc813. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 13 times. W1031 23:03:59.581348 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 13 times. W1031 23:03:59.834357 5816 consensus_peers.cc:357] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 0a8bbcbbb5a6459aa5258464755db3bb (c3-kudu-tst-st02.bj:18700): Couldn't send request to peer 0a8bbcbbb5a6459aa5258464755db3bb for tablet 648eb376255a4dd4bb5818262efcc813. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 14 times. I1031 23:04:00.058825 34067 raft_consensus.cc:411] T 6a813ca355f749b6b31a773aa0f2ac7a P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I1031 23:04:00.058885 34067 raft_consensus.cc:2162] T 6a813ca355f749b6b31a773aa0f2ac7a P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.852s I1031 23:04:00.058940 34067 raft_consensus.cc:435] T 6a813ca355f749b6b31a773aa0f2ac7a P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a8bbcbbb5a6459aa5258464755db3bb" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st02.bj" port: 18700 } } peers { permanent_uuid: "0f53a0d3ef7e44ebb0365c800752d5bd" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st01.bj" port: 18700 } } peers { permanent_uuid: "193181be29284287b25df2ee92938f0b" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st08.bj" port: 18700 } } I1031 23:04:00.059497 34067 leader_election.cc:216] T 6a813ca355f749b6b31a773aa0f2ac7a P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 0a8bbcbbb5a6459aa5258464755db3bb I1031 23:04:00.059528 34067 leader_election.cc:216] T 6a813ca355f749b6b31a773aa0f2ac7a P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 193181be29284287b25df2ee92938f0b W1031 23:04:00.059717 5816 leader_election.cc:278] T 6a813ca355f749b6b31a773aa0f2ac7a P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0a8bbcbbb5a6459aa5258464755db3bb: Illegal state: Tablet not RUNNING: NOT_STARTED W1031 23:04:00.059839 5815 leader_election.cc:278] T 6a813ca355f749b6b31a773aa0f2ac7a P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 193181be29284287b25df2ee92938f0b: Illegal state: Tablet not RUNNING: NOT_STARTED I1031 23:04:00.059857 5815 leader_election.cc:243] T 6a813ca355f749b6b31a773aa0f2ac7a P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. I1031 23:04:00.060187 34254 raft_consensus.cc:2162] T 6a813ca355f749b6b31a773aa0f2ac7a P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 3.221s I1031 23:04:00.060220 34254 raft_consensus.cc:1908] T 6a813ca355f749b6b31a773aa0f2ac7a P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority W1031 23:04:00.081532 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 14 times. W1031 23:04:00.334436 5816 consensus_peers.cc:357] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 0a8bbcbbb5a6459aa5258464755db3bb (c3-kudu-tst-st02.bj:18700): Couldn't send request to peer 0a8bbcbbb5a6459aa5258464755db3bb for tablet 648eb376255a4dd4bb5818262efcc813. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 15 times. W1031 23:04:00.581501 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 15 times. W1031 23:04:00.834502 5816 consensus_peers.cc:357] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 0a8bbcbbb5a6459aa5258464755db3bb (c3-kudu-tst-st02.bj:18700): Couldn't send request to peer 0a8bbcbbb5a6459aa5258464755db3bb for tablet 648eb376255a4dd4bb5818262efcc813. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 16 times. W1031 23:04:01.081648 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 16 times. W1031 23:04:01.334614 5816 consensus_peers.cc:357] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 0a8bbcbbb5a6459aa5258464755db3bb (c3-kudu-tst-st02.bj:18700): Couldn't send request to peer 0a8bbcbbb5a6459aa5258464755db3bb for tablet 648eb376255a4dd4bb5818262efcc813. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 17 times. W1031 23:04:01.581735 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 17 times. W1031 23:04:01.834579 5816 consensus_peers.cc:357] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 0a8bbcbbb5a6459aa5258464755db3bb (c3-kudu-tst-st02.bj:18700): Couldn't send request to peer 0a8bbcbbb5a6459aa5258464755db3bb for tablet 648eb376255a4dd4bb5818262efcc813. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 18 times. W1031 23:04:02.081676 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 18 times. W1031 23:04:02.334709 5816 consensus_peers.cc:357] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 0a8bbcbbb5a6459aa5258464755db3bb (c3-kudu-tst-st02.bj:18700): Couldn't send request to peer 0a8bbcbbb5a6459aa5258464755db3bb for tablet 648eb376255a4dd4bb5818262efcc813. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 19 times. W1031 23:04:02.581820 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 19 times. W1031 23:04:02.834806 5816 consensus_peers.cc:357] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 0a8bbcbbb5a6459aa5258464755db3bb (c3-kudu-tst-st02.bj:18700): Couldn't send request to peer 0a8bbcbbb5a6459aa5258464755db3bb for tablet 648eb376255a4dd4bb5818262efcc813. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 20 times. W1031 23:04:03.081955 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 20 times. W1031 23:04:03.334841 5816 consensus_peers.cc:357] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 0a8bbcbbb5a6459aa5258464755db3bb (c3-kudu-tst-st02.bj:18700): Couldn't send request to peer 0a8bbcbbb5a6459aa5258464755db3bb for tablet 648eb376255a4dd4bb5818262efcc813. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 21 times. W1031 23:04:03.581964 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 21 times. W1031 23:04:03.834925 5816 consensus_peers.cc:357] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 0a8bbcbbb5a6459aa5258464755db3bb (c3-kudu-tst-st02.bj:18700): Couldn't send request to peer 0a8bbcbbb5a6459aa5258464755db3bb for tablet 648eb376255a4dd4bb5818262efcc813. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 22 times. W1031 23:04:04.082224 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 22 times. W1031 23:04:04.335003 5816 consensus_peers.cc:357] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 0a8bbcbbb5a6459aa5258464755db3bb (c3-kudu-tst-st02.bj:18700): Couldn't send request to peer 0a8bbcbbb5a6459aa5258464755db3bb for tablet 648eb376255a4dd4bb5818262efcc813. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 23 times. W1031 23:04:04.582265 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 23 times. W1031 23:04:04.835068 5816 consensus_peers.cc:357] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 0a8bbcbbb5a6459aa5258464755db3bb (c3-kudu-tst-st02.bj:18700): Couldn't send request to peer 0a8bbcbbb5a6459aa5258464755db3bb for tablet 648eb376255a4dd4bb5818262efcc813. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 24 times. W1031 23:04:05.082233 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 24 times. I1031 23:04:05.124168 34067 raft_consensus.cc:411] T 6a813ca355f749b6b31a773aa0f2ac7a P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I1031 23:04:05.124203 34067 raft_consensus.cc:2162] T 6a813ca355f749b6b31a773aa0f2ac7a P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 5.385s I1031 23:04:05.124284 34067 raft_consensus.cc:435] T 6a813ca355f749b6b31a773aa0f2ac7a P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a8bbcbbb5a6459aa5258464755db3bb" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st02.bj" port: 18700 } } peers { permanent_uuid: "0f53a0d3ef7e44ebb0365c800752d5bd" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st01.bj" port: 18700 } } peers { permanent_uuid: "193181be29284287b25df2ee92938f0b" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st08.bj" port: 18700 } } I1031 23:04:05.124852 34067 leader_election.cc:216] T 6a813ca355f749b6b31a773aa0f2ac7a P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 0a8bbcbbb5a6459aa5258464755db3bb I1031 23:04:05.124882 34067 leader_election.cc:216] T 6a813ca355f749b6b31a773aa0f2ac7a P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 193181be29284287b25df2ee92938f0b W1031 23:04:05.125084 5816 leader_election.cc:278] T 6a813ca355f749b6b31a773aa0f2ac7a P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0a8bbcbbb5a6459aa5258464755db3bb: Illegal state: Tablet not RUNNING: NOT_STARTED W1031 23:04:05.125207 5815 leader_election.cc:278] T 6a813ca355f749b6b31a773aa0f2ac7a P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 193181be29284287b25df2ee92938f0b: Illegal state: Tablet not RUNNING: NOT_STARTED I1031 23:04:05.125227 5815 leader_election.cc:243] T 6a813ca355f749b6b31a773aa0f2ac7a P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. I1031 23:04:05.125950 34287 raft_consensus.cc:2162] T 6a813ca355f749b6b31a773aa0f2ac7a P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 7.048s I1031 23:04:05.125980 34287 raft_consensus.cc:1908] T 6a813ca355f749b6b31a773aa0f2ac7a P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority W1031 23:04:05.335157 5816 consensus_peers.cc:357] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 0a8bbcbbb5a6459aa5258464755db3bb (c3-kudu-tst-st02.bj:18700): Couldn't send request to peer 0a8bbcbbb5a6459aa5258464755db3bb for tablet 648eb376255a4dd4bb5818262efcc813. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 25 times. W1031 23:04:05.582461 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 25 times. W1031 23:04:05.835230 5816 consensus_peers.cc:357] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 0a8bbcbbb5a6459aa5258464755db3bb (c3-kudu-tst-st02.bj:18700): Couldn't send request to peer 0a8bbcbbb5a6459aa5258464755db3bb for tablet 648eb376255a4dd4bb5818262efcc813. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 26 times. W1031 23:04:06.082551 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 26 times. W1031 23:04:06.335326 5816 consensus_peers.cc:357] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 0a8bbcbbb5a6459aa5258464755db3bb (c3-kudu-tst-st02.bj:18700): Couldn't send request to peer 0a8bbcbbb5a6459aa5258464755db3bb for tablet 648eb376255a4dd4bb5818262efcc813. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 27 times. W1031 23:04:06.582500 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 27 times. W1031 23:04:06.835384 5816 consensus_peers.cc:357] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 0a8bbcbbb5a6459aa5258464755db3bb (c3-kudu-tst-st02.bj:18700): Couldn't send request to peer 0a8bbcbbb5a6459aa5258464755db3bb for tablet 648eb376255a4dd4bb5818262efcc813. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 28 times. W1031 23:04:07.082702 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 28 times. W1031 23:04:07.335465 5816 consensus_peers.cc:357] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 0a8bbcbbb5a6459aa5258464755db3bb (c3-kudu-tst-st02.bj:18700): Couldn't send request to peer 0a8bbcbbb5a6459aa5258464755db3bb for tablet 648eb376255a4dd4bb5818262efcc813. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 29 times. W1031 23:04:07.582758 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 29 times. I1031 23:04:07.828783 34121 ts_tablet_manager.cc:755] T 42d56cd12bbb4bed9973f96a230fdb9e P 0f53a0d3ef7e44ebb0365c800752d5bd: Time spent starting tablet: real 14.938s user 0.001s sys 0.000s W1031 23:04:07.828816 34121 ts_tablet_manager.cc:785] T 42d56cd12bbb4bed9973f96a230fdb9e P 0f53a0d3ef7e44ebb0365c800752d5bd: Tablet startup took 14937ms W1031 23:04:07.828918 34121 ts_tablet_manager.cc:787] T 42d56cd12bbb4bed9973f96a230fdb9e P 0f53a0d3ef7e44ebb0365c800752d5bd: Trace: 1031 23:03:52.450115 (+ 0us) service_pool.cc:143] Inserting onto call queue 1031 23:03:52.450128 (+ 13us) service_pool.cc:202] Handling call 1031 23:03:52.450337 (+ 209us) ts_tablet_manager.cc:275] Acquired tablet manager lock 1031 23:03:52.450340 (+ 3us) ts_tablet_manager.cc:288] Creating new metadata... 1031 23:03:52.888428 (+438088us) tablet_metadata.cc:461] Metadata flushed 1031 23:03:52.889495 (+ 1067us) inbound_call.cc:130] Queueing success response 1031 23:03:52.889519 (+ 24us) ts_tablet_manager.cc:724] Bootstrapping tablet 1031 23:03:52.891248 (+ 1729us) ts_tablet_manager.cc:756] Initializing tablet peer 1031 23:03:52.891266 (+ 18us) tablet_peer.cc:161] Creating consensus instance 1031 23:03:52.891527 (+ 261us) tablet_peer.cc:183] Starting instrumentation 1031 23:03:52.891530 (+ 3us) tablet_peer.cc:188] TabletPeer::Init() finished 1031 23:03:52.891530 (+ 0us) ts_tablet_manager.cc:771] Starting tablet peer 1031 23:03:52.891531 (+ 1us) tablet_peer.cc:195] Starting consensus Metrics: {"fdatasync":1,"fdatasync_us":144487,"mutex_wait_us":14936592,"raft.queue_time_us":351,"raft.run_cpu_time_us":3,"raft.run_wall_time_us":3,"tablet-open.queue_time_us":425,"tcmalloc_contention_cycles":123904,"thread_start_us":850,"threads_started":3} I1031 23:04:07.828930 34047 ts_tablet_manager.cc:755] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd: Time spent starting tablet: real 15.044s user 0.002s sys 0.000s W1031 23:04:07.828959 34047 ts_tablet_manager.cc:785] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd: Tablet startup took 15043ms I1031 23:04:07.829057 34087 ts_tablet_manager.cc:755] T e003f6e546374517aa9a51b4bc23c827 P 0f53a0d3ef7e44ebb0365c800752d5bd: Time spent starting tablet: real 14.963s user 0.001s sys 0.000s W1031 23:04:07.829061 34047 ts_tablet_manager.cc:787] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd: Trace: 1031 23:03:52.450213 (+ 0us) service_pool.cc:143] Inserting onto call queue 1031 23:03:52.450225 (+ 12us) service_pool.cc:202] Handling call 1031 23:03:52.450281 (+ 56us) ts_tablet_manager.cc:275] Acquired tablet manager lock 1031 23:03:52.450283 (+ 2us) ts_tablet_manager.cc:288] Creating new metadata... 1031 23:03:52.770767 (+320484us) tablet_metadata.cc:461] Metadata flushed 1031 23:03:52.771935 (+ 1168us) inbound_call.cc:130] Queueing success response 1031 23:03:52.771999 (+ 64us) ts_tablet_manager.cc:724] Bootstrapping tablet 1031 23:03:52.785342 (+ 13343us) ts_tablet_manager.cc:756] Initializing tablet peer 1031 23:03:52.785377 (+ 35us) tablet_peer.cc:161] Creating consensus instance 1031 23:03:52.785737 (+ 360us) tablet_peer.cc:183] Starting instrumentation 1031 23:03:52.785743 (+ 6us) tablet_peer.cc:188] TabletPeer::Init() finished 1031 23:03:52.785744 (+ 1us) ts_tablet_manager.cc:771] Starting tablet peer 1031 23:03:52.785744 (+ 0us) tablet_peer.cc:195] Starting consensus Metrics: {"fdatasync":1,"fdatasync_us":144743,"mutex_wait_us":15042196,"raft.queue_time_us":361,"raft.run_cpu_time_us":4,"raft.run_wall_time_us":4,"tablet-open.queue_time_us":504,"thread_start_us":1206,"threads_started":3} W1031 23:04:07.829083 34087 ts_tablet_manager.cc:785] T e003f6e546374517aa9a51b4bc23c827 P 0f53a0d3ef7e44ebb0365c800752d5bd: Tablet startup took 14963ms I1031 23:04:07.829222 34062 ts_tablet_manager.cc:755] T 6a813ca355f749b6b31a773aa0f2ac7a P 0f53a0d3ef7e44ebb0365c800752d5bd: Time spent starting tablet: real 14.988s user 0.000s sys 0.001s W1031 23:04:07.829252 34062 ts_tablet_manager.cc:785] T 6a813ca355f749b6b31a773aa0f2ac7a P 0f53a0d3ef7e44ebb0365c800752d5bd: Tablet startup took 14987ms W1031 23:04:07.829289 34087 ts_tablet_manager.cc:787] T e003f6e546374517aa9a51b4bc23c827 P 0f53a0d3ef7e44ebb0365c800752d5bd: Trace: 1031 23:03:52.449679 (+ 0us) service_pool.cc:143] Inserting onto call queue 1031 23:03:52.449691 (+ 12us) service_pool.cc:202] Handling call 1031 23:03:52.449893 (+ 202us) ts_tablet_manager.cc:275] Acquired tablet manager lock 1031 23:03:52.449896 (+ 3us) ts_tablet_manager.cc:288] Creating new metadata... 1031 23:03:52.863146 (+413250us) tablet_metadata.cc:461] Metadata flushed 1031 23:03:52.864134 (+ 988us) inbound_call.cc:130] Queueing success response 1031 23:03:52.864167 (+ 33us) ts_tablet_manager.cc:724] Bootstrapping tablet 1031 23:03:52.865894 (+ 1727us) ts_tablet_manager.cc:756] Initializing tablet peer 1031 23:03:52.865909 (+ 15us) tablet_peer.cc:161] Creating consensus instance 1031 23:03:52.866189 (+ 280us) tablet_peer.cc:183] Starting instrumentation 1031 23:03:52.866192 (+ 3us) tablet_peer.cc:188] TabletPeer::Init() finished 1031 23:03:52.866192 (+ 0us) ts_tablet_manager.cc:771] Starting tablet peer 1031 23:03:52.866193 (+ 1us) tablet_peer.cc:195] Starting consensus Metrics: {"fdatasync":1,"fdatasync_us":144877,"mutex_wait_us":14962184,"raft.queue_time_us":391,"raft.run_cpu_time_us":7,"raft.run_wall_time_us":6,"tablet-open.queue_time_us":467,"tcmalloc_contention_cycles":69888,"thread_start_us":931,"threads_started":3} I1031 23:04:07.829339 34052 ts_tablet_manager.cc:755] T 865b2d5788f44598b169ca6415314610 P 0f53a0d3ef7e44ebb0365c800752d5bd: Time spent starting tablet: real 15.014s user 0.000s sys 0.001s W1031 23:04:07.829355 34062 ts_tablet_manager.cc:787] T 6a813ca355f749b6b31a773aa0f2ac7a P 0f53a0d3ef7e44ebb0365c800752d5bd: Trace: 1031 23:03:52.450004 (+ 0us) service_pool.cc:143] Inserting onto call queue 1031 23:03:52.450043 (+ 39us) service_pool.cc:202] Handling call 1031 23:03:52.450188 (+ 145us) ts_tablet_manager.cc:275] Acquired tablet manager lock 1031 23:03:52.450191 (+ 3us) ts_tablet_manager.cc:288] Creating new metadata... 1031 23:03:52.837872 (+387681us) tablet_metadata.cc:461] Metadata flushed 1031 23:03:52.838887 (+ 1015us) inbound_call.cc:130] Queueing success response 1031 23:03:52.838935 (+ 48us) ts_tablet_manager.cc:724] Bootstrapping tablet 1031 23:03:52.841379 (+ 2444us) ts_tablet_manager.cc:756] Initializing tablet peer 1031 23:03:52.841394 (+ 15us) tablet_peer.cc:161] Creating consensus instance 1031 23:03:52.841657 (+ 263us) tablet_peer.cc:183] Starting instrumentation 1031 23:03:52.841660 (+ 3us) tablet_peer.cc:188] TabletPeer::Init() finished 1031 23:03:52.841661 (+ 1us) ts_tablet_manager.cc:771] Starting tablet peer 1031 23:03:52.841661 (+ 0us) tablet_peer.cc:195] Starting consensus Metrics: {"fdatasync":1,"fdatasync_us":144596,"mutex_wait_us":14986636,"raft.queue_time_us":360,"raft.run_cpu_time_us":3,"raft.run_wall_time_us":3,"tablet-open.queue_time_us":467,"tcmalloc_contention_cycles":121088,"thread_start_us":1166,"threads_started":3} I1031 23:04:07.829366 34126 ts_tablet_manager.cc:755] T de45d05c3a064dc5acb91d66dcaea6ed P 0f53a0d3ef7e44ebb0365c800752d5bd: Time spent starting tablet: real 14.913s user 0.001s sys 0.000s W1031 23:04:07.829357 34052 ts_tablet_manager.cc:785] T 865b2d5788f44598b169ca6415314610 P 0f53a0d3ef7e44ebb0365c800752d5bd: Tablet startup took 15013ms W1031 23:04:07.829396 34126 ts_tablet_manager.cc:785] T de45d05c3a064dc5acb91d66dcaea6ed P 0f53a0d3ef7e44ebb0365c800752d5bd: Tablet startup took 14913ms I1031 23:04:07.829406 34134 ts_tablet_manager.cc:755] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd: Time spent starting tablet: real 14.880s user 0.001s sys 0.000s W1031 23:04:07.829427 34134 ts_tablet_manager.cc:785] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd: Tablet startup took 14880ms W1031 23:04:07.829560 34052 ts_tablet_manager.cc:787] T 865b2d5788f44598b169ca6415314610 P 0f53a0d3ef7e44ebb0365c800752d5bd: Trace: 1031 23:03:52.449719 (+ 0us) service_pool.cc:143] Inserting onto call queue 1031 23:03:52.449734 (+ 15us) service_pool.cc:202] Handling call 1031 23:03:52.449852 (+ 118us) ts_tablet_manager.cc:275] Acquired tablet manager lock 1031 23:03:52.449856 (+ 4us) ts_tablet_manager.cc:288] Creating new metadata... 1031 23:03:52.812698 (+362842us) tablet_metadata.cc:461] Metadata flushed 1031 23:03:52.813466 (+ 768us) inbound_call.cc:130] Queueing success response 1031 23:03:52.813488 (+ 22us) ts_tablet_manager.cc:724] Bootstrapping tablet 1031 23:03:52.815512 (+ 2024us) ts_tablet_manager.cc:756] Initializing tablet peer 1031 23:03:52.815528 (+ 16us) tablet_peer.cc:161] Creating consensus instance 1031 23:03:52.815792 (+ 264us) tablet_peer.cc:183] Starting instrumentation 1031 23:03:52.815795 (+ 3us) tablet_peer.cc:188] TabletPeer::Init() finished 1031 23:03:52.815796 (+ 1us) ts_tablet_manager.cc:771] Starting tablet peer 1031 23:03:52.815797 (+ 1us) tablet_peer.cc:195] Starting consensus Metrics: {"fdatasync":1,"fdatasync_us":144739,"mutex_wait_us":15012832,"raft.queue_time_us":139,"raft.run_cpu_time_us":22,"raft.run_wall_time_us":22,"tablet-open.queue_time_us":123,"tcmalloc_contention_cycles":896,"thread_start_us":597,"threads_started":3} W1031 23:04:07.829572 34126 ts_tablet_manager.cc:787] T de45d05c3a064dc5acb91d66dcaea6ed P 0f53a0d3ef7e44ebb0365c800752d5bd: Trace: 1031 23:03:52.449694 (+ 0us) service_pool.cc:143] Inserting onto call queue 1031 23:03:52.449717 (+ 23us) service_pool.cc:202] Handling call 1031 23:03:52.449888 (+ 171us) ts_tablet_manager.cc:275] Acquired tablet manager lock 1031 23:03:52.449892 (+ 4us) ts_tablet_manager.cc:288] Creating new metadata... 1031 23:03:52.913575 (+463683us) tablet_metadata.cc:461] Metadata flushed 1031 23:03:52.914521 (+ 946us) inbound_call.cc:130] Queueing success response 1031 23:03:52.914552 (+ 31us) ts_tablet_manager.cc:724] Bootstrapping tablet 1031 23:03:52.916057 (+ 1505us) ts_tablet_manager.cc:756] Initializing tablet peer 1031 23:03:52.916073 (+ 16us) tablet_peer.cc:161] Creating consensus instance 1031 23:03:52.916372 (+ 299us) tablet_peer.cc:183] Starting instrumentation 1031 23:03:52.916376 (+ 4us) tablet_peer.cc:188] TabletPeer::Init() finished 1031 23:03:52.916377 (+ 1us) ts_tablet_manager.cc:771] Starting tablet peer 1031 23:03:52.916377 (+ 0us) tablet_peer.cc:195] Starting consensus Metrics: {"fdatasync":1,"fdatasync_us":144971,"mutex_wait_us":14912325,"raft.queue_time_us":105,"raft.run_cpu_time_us":3,"raft.run_wall_time_us":3,"tablet-open.queue_time_us":455,"tcmalloc_contention_cycles":145664,"thread_start_us":913,"threads_started":3} W1031 23:04:07.829576 34134 ts_tablet_manager.cc:787] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd: Trace: 1031 23:03:52.449665 (+ 0us) service_pool.cc:143] Inserting onto call queue 1031 23:03:52.449677 (+ 12us) service_pool.cc:202] Handling call 1031 23:03:52.449825 (+ 148us) ts_tablet_manager.cc:275] Acquired tablet manager lock 1031 23:03:52.449829 (+ 4us) ts_tablet_manager.cc:288] Creating new metadata... 1031 23:03:52.947149 (+497320us) tablet_metadata.cc:461] Metadata flushed 1031 23:03:52.948035 (+ 886us) inbound_call.cc:130] Queueing success response 1031 23:03:52.948066 (+ 31us) ts_tablet_manager.cc:724] Bootstrapping tablet 1031 23:03:52.949121 (+ 1055us) ts_tablet_manager.cc:756] Initializing tablet peer 1031 23:03:52.949137 (+ 16us) tablet_peer.cc:161] Creating consensus instance 1031 23:03:52.949408 (+ 271us) tablet_peer.cc:183] Starting instrumentation 1031 23:03:52.949421 (+ 13us) tablet_peer.cc:188] TabletPeer::Init() finished 1031 23:03:52.949421 (+ 0us) ts_tablet_manager.cc:771] Starting tablet peer 1031 23:03:52.949422 (+ 1us) tablet_peer.cc:195] Starting consensus Metrics: {"fdatasync":1,"fdatasync_us":145023,"mutex_wait_us":14879214,"raft.queue_time_us":460,"raft.run_cpu_time_us":3,"raft.run_wall_time_us":3,"tablet-open.queue_time_us":454,"tcmalloc_contention_cycles":48896,"thread_start_us":1002,"threads_started":3} W1031 23:04:07.835527 5816 consensus_peers.cc:357] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 0a8bbcbbb5a6459aa5258464755db3bb (c3-kudu-tst-st02.bj:18700): Couldn't send request to peer 0a8bbcbbb5a6459aa5258464755db3bb for tablet 648eb376255a4dd4bb5818262efcc813. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 30 times. I1031 23:04:07.895853 5734 maintenance_manager.cc:419] Time spent running UndoDeltaBlockGCOp(b0174e1a99694688bdc932c6a00233ce): real 0.067s user 0.002s sys 0.000s I1031 23:04:07.895990 5734 maintenance_manager.cc:425] P 0f53a0d3ef7e44ebb0365c800752d5bd: UndoDeltaBlockGCOp(b0174e1a99694688bdc932c6a00233ce) metrics: {"data dir 6.queue_time_us":224,"data dir 6.run_cpu_time_us":72,"data dir 6.run_wall_time_us":72,"fdatasync":1,"fdatasync_us":40284,"thread_start_us":210,"threads_started":1} W1031 23:04:08.082794 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 30 times. W1031 23:04:08.335655 5816 consensus_peers.cc:357] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 0a8bbcbbb5a6459aa5258464755db3bb (c3-kudu-tst-st02.bj:18700): Couldn't send request to peer 0a8bbcbbb5a6459aa5258464755db3bb for tablet 648eb376255a4dd4bb5818262efcc813. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 31 times. I1031 23:04:08.435953 5734 maintenance_manager.cc:419] Time spent running UndoDeltaBlockGCOp(3409046e569643c3a083c3f27cfe1582): real 0.205s user 0.147s sys 0.003s I1031 23:04:08.436233 5734 maintenance_manager.cc:425] P 0f53a0d3ef7e44ebb0365c800752d5bd: UndoDeltaBlockGCOp(3409046e569643c3a083c3f27cfe1582) metrics: {"cfile_init":1,"data dir 1.queue_time_us":204,"data dir 1.run_cpu_time_us":52,"data dir 1.run_wall_time_us":53,"fdatasync":1,"fdatasync_us":39201,"lbm_read_time_us":14,"lbm_reads_lt_1ms":4,"tcmalloc_contention_cycles":70400,"thread_start_us":164,"threads_started":1} W1031 23:04:08.582957 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 31 times. I1031 23:04:08.647860 5734 maintenance_manager.cc:419] Time spent running UndoDeltaBlockGCOp(0c685a7574c64d9dbae7fb02787cc85e): real 0.015s user 0.014s sys 0.000s I1031 23:04:08.647918 5734 maintenance_manager.cc:425] P 0f53a0d3ef7e44ebb0365c800752d5bd: UndoDeltaBlockGCOp(0c685a7574c64d9dbae7fb02787cc85e) metrics: {"cfile_init":1,"lbm_read_time_us":13,"lbm_reads_lt_1ms":4} W1031 23:04:08.835690 5816 consensus_peers.cc:357] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 0a8bbcbbb5a6459aa5258464755db3bb (c3-kudu-tst-st02.bj:18700): Couldn't send request to peer 0a8bbcbbb5a6459aa5258464755db3bb for tablet 648eb376255a4dd4bb5818262efcc813. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 32 times. I1031 23:04:09.037394 5734 delta_tracker.cc:686] T 3409046e569643c3a083c3f27cfe1582 P 0f53a0d3ef7e44ebb0365c800752d5bd: Flushing 1 deltas from DMS 1... I1031 23:04:09.070906 5734 delta_tracker.cc:628] T 3409046e569643c3a083c3f27cfe1582 P 0f53a0d3ef7e44ebb0365c800752d5bd: Flushed delta block: 0000000044990269 ts range: [6182756987042463744, 6182756987042463744] I1031 23:04:09.070936 5734 delta_tracker.cc:641] T 3409046e569643c3a083c3f27cfe1582 P 0f53a0d3ef7e44ebb0365c800752d5bd: Reopened delta block for read: 0000000044990269 W1031 23:04:09.082780 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 32 times. I1031 23:04:09.247954 5734 maintenance_manager.cc:419] Time spent running FlushDeltaMemStoresOp(3409046e569643c3a083c3f27cfe1582): real 0.212s user 0.114s sys 0.005s I1031 23:04:09.248023 5734 maintenance_manager.cc:425] P 0f53a0d3ef7e44ebb0365c800752d5bd: FlushDeltaMemStoresOp(3409046e569643c3a083c3f27cfe1582) metrics: {"fdatasync":3,"fdatasync_us":76909,"lbm_write_time_us":58,"lbm_writes_lt_1ms":6,"tcmalloc_contention_cycles":194176} W1031 23:04:09.335814 5816 consensus_peers.cc:357] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 0a8bbcbbb5a6459aa5258464755db3bb (c3-kudu-tst-st02.bj:18700): Couldn't send request to peer 0a8bbcbbb5a6459aa5258464755db3bb for tablet 648eb376255a4dd4bb5818262efcc813. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 33 times. W1031 23:04:09.583168 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 33 times. I1031 23:04:09.714359 5734 tablet.cc:911] T c62e1dbe04164d22bb6fdf7674bc18c5 P 0f53a0d3ef7e44ebb0365c800752d5bd: Flush: entering stage 1 (old memrowset already frozen for inserts) I1031 23:04:09.714417 5734 compaction.cc:877] Selected 1 rowsets to compact: I1031 23:04:09.714432 5734 compaction.cc:880] memrowset(current size on disk: ~0 bytes) I1031 23:04:09.714445 5734 tablet.cc:913] T c62e1dbe04164d22bb6fdf7674bc18c5 P 0f53a0d3ef7e44ebb0365c800752d5bd: Memstore in-memory size: 3751 bytes I1031 23:04:09.714478 5734 tablet.cc:1174] T c62e1dbe04164d22bb6fdf7674bc18c5 P 0f53a0d3ef7e44ebb0365c800752d5bd: Flush: entering phase 1 (flushing snapshot). Phase 1 snapshot: MvccSnapshot[committed={T|T < 6182757230385729536 or (T in {6182757230385729536})}] I1031 23:04:09.714632 5734 multi_column_writer.cc:85] Opened CFile writer for column ts[int32 NOT NULL] I1031 23:04:09.714670 5734 multi_column_writer.cc:85] Opened CFile writer for column id[int32 NOT NULL] W1031 23:04:09.835888 5816 consensus_peers.cc:357] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 0a8bbcbbb5a6459aa5258464755db3bb (c3-kudu-tst-st02.bj:18700): Couldn't send request to peer 0a8bbcbbb5a6459aa5258464755db3bb for tablet 648eb376255a4dd4bb5818262efcc813. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 34 times. I1031 23:04:09.862049 5734 tablet.cc:1259] T c62e1dbe04164d22bb6fdf7674bc18c5 P 0f53a0d3ef7e44ebb0365c800752d5bd: Flush: entering phase 2 (starting to duplicate updates in new rowsets) W1031 23:04:10.083168 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 34 times. I1031 23:04:10.093739 5734 tablet.cc:1314] T c62e1dbe04164d22bb6fdf7674bc18c5 P 0f53a0d3ef7e44ebb0365c800752d5bd: Flush Phase 2: carrying over any updates which arrived during Phase 1 I1031 23:04:10.093837 5734 tablet.cc:1316] T c62e1dbe04164d22bb6fdf7674bc18c5 P 0f53a0d3ef7e44ebb0365c800752d5bd: Phase 2 snapshot: MvccSnapshot[committed={T|T < 6182757230385729536 or (T in {6182757230385729536})}] W1031 23:04:10.336083 5816 consensus_peers.cc:357] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 0a8bbcbbb5a6459aa5258464755db3bb (c3-kudu-tst-st02.bj:18700): Couldn't send request to peer 0a8bbcbbb5a6459aa5258464755db3bb for tablet 648eb376255a4dd4bb5818262efcc813. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 35 times. I1031 23:04:10.502933 5734 tablet.cc:1358] T c62e1dbe04164d22bb6fdf7674bc18c5 P 0f53a0d3ef7e44ebb0365c800752d5bd: Flush successful on 16 rows (4331 bytes) I1031 23:04:10.503013 5734 maintenance_manager.cc:419] Time spent running FlushMRSOp(c62e1dbe04164d22bb6fdf7674bc18c5): real 1.021s user 0.814s sys 0.004s I1031 23:04:10.503182 5734 maintenance_manager.cc:425] P 0f53a0d3ef7e44ebb0365c800752d5bd: FlushMRSOp(c62e1dbe04164d22bb6fdf7674bc18c5) metrics: {"cfile_init":1,"fdatasync":11,"fdatasync_us":186986,"lbm_read_time_us":6,"lbm_reads_lt_1ms":4,"lbm_write_time_us":169,"lbm_writes_lt_1ms":26,"tcmalloc_contention_cycles":114944} W1031 23:04:10.583269 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 35 times. W1031 23:04:10.836017 5816 consensus_peers.cc:357] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 0a8bbcbbb5a6459aa5258464755db3bb (c3-kudu-tst-st02.bj:18700): Couldn't send request to peer 0a8bbcbbb5a6459aa5258464755db3bb for tablet 648eb376255a4dd4bb5818262efcc813. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 36 times. W1031 23:04:11.083227 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 36 times. W1031 23:04:11.336093 5816 consensus_peers.cc:357] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 0a8bbcbbb5a6459aa5258464755db3bb (c3-kudu-tst-st02.bj:18700): Couldn't send request to peer 0a8bbcbbb5a6459aa5258464755db3bb for tablet 648eb376255a4dd4bb5818262efcc813. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 37 times. W1031 23:04:11.583412 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 37 times. W1031 23:04:11.836109 5816 consensus_peers.cc:357] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 0a8bbcbbb5a6459aa5258464755db3bb (c3-kudu-tst-st02.bj:18700): Couldn't send request to peer 0a8bbcbbb5a6459aa5258464755db3bb for tablet 648eb376255a4dd4bb5818262efcc813. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 38 times. W1031 23:04:12.083434 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 38 times. W1031 23:04:12.336179 5816 consensus_peers.cc:357] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 0a8bbcbbb5a6459aa5258464755db3bb (c3-kudu-tst-st02.bj:18700): Couldn't send request to peer 0a8bbcbbb5a6459aa5258464755db3bb for tablet 648eb376255a4dd4bb5818262efcc813. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 39 times. W1031 23:04:12.583555 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 39 times. W1031 23:04:12.836290 5816 consensus_peers.cc:357] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 0a8bbcbbb5a6459aa5258464755db3bb (c3-kudu-tst-st02.bj:18700): Couldn't send request to peer 0a8bbcbbb5a6459aa5258464755db3bb for tablet 648eb376255a4dd4bb5818262efcc813. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 40 times. W1031 23:04:13.083504 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 40 times. W1031 23:04:13.336405 5816 consensus_peers.cc:357] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 0a8bbcbbb5a6459aa5258464755db3bb (c3-kudu-tst-st02.bj:18700): Couldn't send request to peer 0a8bbcbbb5a6459aa5258464755db3bb for tablet 648eb376255a4dd4bb5818262efcc813. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 41 times. W1031 23:04:13.583631 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 41 times. W1031 23:04:13.836390 5816 consensus_peers.cc:357] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 0a8bbcbbb5a6459aa5258464755db3bb (c3-kudu-tst-st02.bj:18700): Couldn't send request to peer 0a8bbcbbb5a6459aa5258464755db3bb for tablet 648eb376255a4dd4bb5818262efcc813. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 42 times. W1031 23:04:14.083727 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 42 times. I1031 23:04:14.184551 34067 raft_consensus.cc:411] T 6a813ca355f749b6b31a773aa0f2ac7a P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I1031 23:04:14.184597 34067 raft_consensus.cc:2162] T 6a813ca355f749b6b31a773aa0f2ac7a P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 2.795s I1031 23:04:14.184649 34067 raft_consensus.cc:435] T 6a813ca355f749b6b31a773aa0f2ac7a P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a8bbcbbb5a6459aa5258464755db3bb" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st02.bj" port: 18700 } } peers { permanent_uuid: "0f53a0d3ef7e44ebb0365c800752d5bd" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st01.bj" port: 18700 } } peers { permanent_uuid: "193181be29284287b25df2ee92938f0b" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st08.bj" port: 18700 } } I1031 23:04:14.185187 34067 leader_election.cc:216] T 6a813ca355f749b6b31a773aa0f2ac7a P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 0a8bbcbbb5a6459aa5258464755db3bb I1031 23:04:14.185219 34067 leader_election.cc:216] T 6a813ca355f749b6b31a773aa0f2ac7a P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 193181be29284287b25df2ee92938f0b W1031 23:04:14.185454 5816 leader_election.cc:278] T 6a813ca355f749b6b31a773aa0f2ac7a P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0a8bbcbbb5a6459aa5258464755db3bb: Illegal state: Tablet not RUNNING: NOT_STARTED W1031 23:04:14.185554 5815 leader_election.cc:278] T 6a813ca355f749b6b31a773aa0f2ac7a P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 193181be29284287b25df2ee92938f0b: Illegal state: Tablet not RUNNING: NOT_STARTED I1031 23:04:14.185582 5815 leader_election.cc:243] T 6a813ca355f749b6b31a773aa0f2ac7a P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. I1031 23:04:14.185838 34319 raft_consensus.cc:2162] T 6a813ca355f749b6b31a773aa0f2ac7a P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 3.033s I1031 23:04:14.185855 34319 raft_consensus.cc:1908] T 6a813ca355f749b6b31a773aa0f2ac7a P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority W1031 23:04:14.336494 5816 consensus_peers.cc:357] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 0a8bbcbbb5a6459aa5258464755db3bb (c3-kudu-tst-st02.bj:18700): Couldn't send request to peer 0a8bbcbbb5a6459aa5258464755db3bb for tablet 648eb376255a4dd4bb5818262efcc813. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 43 times. W1031 23:04:14.583883 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 43 times. W1031 23:04:14.836575 5816 consensus_peers.cc:357] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 0a8bbcbbb5a6459aa5258464755db3bb (c3-kudu-tst-st02.bj:18700): Couldn't send request to peer 0a8bbcbbb5a6459aa5258464755db3bb for tablet 648eb376255a4dd4bb5818262efcc813. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 44 times. W1031 23:04:15.083969 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 44 times. W1031 23:04:15.336776 5816 consensus_peers.cc:357] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 0a8bbcbbb5a6459aa5258464755db3bb (c3-kudu-tst-st02.bj:18700): Couldn't send request to peer 0a8bbcbbb5a6459aa5258464755db3bb for tablet 648eb376255a4dd4bb5818262efcc813. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 45 times. W1031 23:04:15.584137 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 45 times. W1031 23:04:15.836760 5816 consensus_peers.cc:357] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 0a8bbcbbb5a6459aa5258464755db3bb (c3-kudu-tst-st02.bj:18700): Couldn't send request to peer 0a8bbcbbb5a6459aa5258464755db3bb for tablet 648eb376255a4dd4bb5818262efcc813. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 46 times. W1031 23:04:16.084152 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 46 times. W1031 23:04:16.336961 5816 consensus_peers.cc:357] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 0a8bbcbbb5a6459aa5258464755db3bb (c3-kudu-tst-st02.bj:18700): Couldn't send request to peer 0a8bbcbbb5a6459aa5258464755db3bb for tablet 648eb376255a4dd4bb5818262efcc813. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 47 times. W1031 23:04:16.584273 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 47 times. W1031 23:04:16.836894 5816 consensus_peers.cc:357] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 0a8bbcbbb5a6459aa5258464755db3bb (c3-kudu-tst-st02.bj:18700): Couldn't send request to peer 0a8bbcbbb5a6459aa5258464755db3bb for tablet 648eb376255a4dd4bb5818262efcc813. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 48 times. W1031 23:04:17.084383 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 48 times. W1031 23:04:17.337004 5816 consensus_peers.cc:357] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 0a8bbcbbb5a6459aa5258464755db3bb (c3-kudu-tst-st02.bj:18700): Couldn't send request to peer 0a8bbcbbb5a6459aa5258464755db3bb for tablet 648eb376255a4dd4bb5818262efcc813. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 49 times. W1031 23:04:17.584347 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 49 times. W1031 23:04:17.837041 5816 consensus_peers.cc:357] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 0a8bbcbbb5a6459aa5258464755db3bb (c3-kudu-tst-st02.bj:18700): Couldn't send request to peer 0a8bbcbbb5a6459aa5258464755db3bb for tablet 648eb376255a4dd4bb5818262efcc813. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 50 times. W1031 23:04:18.084480 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 50 times. W1031 23:04:18.337245 5816 consensus_peers.cc:357] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 0a8bbcbbb5a6459aa5258464755db3bb (c3-kudu-tst-st02.bj:18700): Couldn't send request to peer 0a8bbcbbb5a6459aa5258464755db3bb for tablet 648eb376255a4dd4bb5818262efcc813. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 51 times. W1031 23:04:18.584676 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 51 times. W1031 23:04:18.837183 5816 consensus_peers.cc:357] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 0a8bbcbbb5a6459aa5258464755db3bb (c3-kudu-tst-st02.bj:18700): Couldn't send request to peer 0a8bbcbbb5a6459aa5258464755db3bb for tablet 648eb376255a4dd4bb5818262efcc813. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 52 times. W1031 23:04:19.084758 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 52 times. I1031 23:04:19.091919 34067 raft_consensus.cc:411] T 6a813ca355f749b6b31a773aa0f2ac7a P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I1031 23:04:19.091953 34067 raft_consensus.cc:2162] T 6a813ca355f749b6b31a773aa0f2ac7a P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 6.094s I1031 23:04:19.092005 34067 raft_consensus.cc:435] T 6a813ca355f749b6b31a773aa0f2ac7a P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a8bbcbbb5a6459aa5258464755db3bb" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st02.bj" port: 18700 } } peers { permanent_uuid: "0f53a0d3ef7e44ebb0365c800752d5bd" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st01.bj" port: 18700 } } peers { permanent_uuid: "193181be29284287b25df2ee92938f0b" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st08.bj" port: 18700 } } I1031 23:04:19.092571 34067 leader_election.cc:216] T 6a813ca355f749b6b31a773aa0f2ac7a P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 0a8bbcbbb5a6459aa5258464755db3bb I1031 23:04:19.092600 34067 leader_election.cc:216] T 6a813ca355f749b6b31a773aa0f2ac7a P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 193181be29284287b25df2ee92938f0b W1031 23:04:19.092839 5816 leader_election.cc:278] T 6a813ca355f749b6b31a773aa0f2ac7a P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0a8bbcbbb5a6459aa5258464755db3bb: Illegal state: Tablet not RUNNING: NOT_STARTED W1031 23:04:19.092891 5815 leader_election.cc:278] T 6a813ca355f749b6b31a773aa0f2ac7a P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 193181be29284287b25df2ee92938f0b: Illegal state: Tablet not RUNNING: NOT_STARTED I1031 23:04:19.092923 5815 leader_election.cc:243] T 6a813ca355f749b6b31a773aa0f2ac7a P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. I1031 23:04:19.093240 34553 raft_consensus.cc:2162] T 6a813ca355f749b6b31a773aa0f2ac7a P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 16.370s I1031 23:04:19.093268 34553 raft_consensus.cc:1908] T 6a813ca355f749b6b31a773aa0f2ac7a P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority W1031 23:04:19.337339 5816 consensus_peers.cc:357] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 0a8bbcbbb5a6459aa5258464755db3bb (c3-kudu-tst-st02.bj:18700): Couldn't send request to peer 0a8bbcbbb5a6459aa5258464755db3bb for tablet 648eb376255a4dd4bb5818262efcc813. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 53 times. W1031 23:04:19.584815 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 53 times. W1031 23:04:19.837365 5816 consensus_peers.cc:357] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 0a8bbcbbb5a6459aa5258464755db3bb (c3-kudu-tst-st02.bj:18700): Couldn't send request to peer 0a8bbcbbb5a6459aa5258464755db3bb for tablet 648eb376255a4dd4bb5818262efcc813. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 54 times. W1031 23:04:20.084831 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 54 times. W1031 23:04:20.337448 5816 consensus_peers.cc:357] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 0a8bbcbbb5a6459aa5258464755db3bb (c3-kudu-tst-st02.bj:18700): Couldn't send request to peer 0a8bbcbbb5a6459aa5258464755db3bb for tablet 648eb376255a4dd4bb5818262efcc813. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 55 times. W1031 23:04:20.584884 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 55 times. W1031 23:04:20.837477 5816 consensus_peers.cc:357] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 0a8bbcbbb5a6459aa5258464755db3bb (c3-kudu-tst-st02.bj:18700): Couldn't send request to peer 0a8bbcbbb5a6459aa5258464755db3bb for tablet 648eb376255a4dd4bb5818262efcc813. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 56 times. W1031 23:04:21.084952 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 56 times. W1031 23:04:21.337574 5816 consensus_peers.cc:357] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 0a8bbcbbb5a6459aa5258464755db3bb (c3-kudu-tst-st02.bj:18700): Couldn't send request to peer 0a8bbcbbb5a6459aa5258464755db3bb for tablet 648eb376255a4dd4bb5818262efcc813. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 57 times. W1031 23:04:21.584944 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 57 times. W1031 23:04:21.837610 5816 consensus_peers.cc:357] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 0a8bbcbbb5a6459aa5258464755db3bb (c3-kudu-tst-st02.bj:18700): Couldn't send request to peer 0a8bbcbbb5a6459aa5258464755db3bb for tablet 648eb376255a4dd4bb5818262efcc813. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 58 times. W1031 23:04:22.085161 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 58 times. W1031 23:04:22.337754 5816 consensus_peers.cc:357] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 0a8bbcbbb5a6459aa5258464755db3bb (c3-kudu-tst-st02.bj:18700): Couldn't send request to peer 0a8bbcbbb5a6459aa5258464755db3bb for tablet 648eb376255a4dd4bb5818262efcc813. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 59 times. I1031 23:04:22.476685 5853 tablet_service.cc:691] Processing DeleteTablet for tablet 6a813ca355f749b6b31a773aa0f2ac7a with delete_type TABLET_DATA_DELETED (Replaced by 582c1281bf5147b2bd40c9a950a1152f at 2017-10-31 23:04:22 CST) from {username='work'} at 10.108.187.10:53997 I1031 23:04:22.476716 5853 tablet_peer.cc:221] Initiating TabletPeer shutdown for tablet: 6a813ca355f749b6b31a773aa0f2ac7a I1031 23:04:22.477208 5869 tablet_service.cc:650] Processing CreateTablet for tablet bff7dab34ccd4b42842077e82106f6fb (table=default.timeout_tst [id=cdcb3e60c1314776843d856365535d49]), partition=HASH (id) PARTITION 0, RANGE (date) PARTITION 20171028 <= VALUES < 20171029 I1031 23:04:22.477303 5866 tablet_service.cc:650] Processing CreateTablet for tablet 117cd74bcdab4942901acb0005cdd702 (table=default.timeout_tst [id=cdcb3e60c1314776843d856365535d49]), partition=HASH (id) PARTITION 1, RANGE (date) PARTITION 20171026 <= VALUES < 20171027 I1031 23:04:22.477326 5857 tablet_service.cc:650] Processing CreateTablet for tablet 56155a2adc24450d89e338cc955bfa18 (table=default.timeout_tst [id=cdcb3e60c1314776843d856365535d49]), partition=HASH (id) PARTITION 0, RANGE (date) PARTITION 20171029 <= VALUES < 20171030 I1031 23:04:22.477679 5861 tablet_service.cc:650] Processing CreateTablet for tablet 991306ec7acb449c89e5dd29d90e8d55 (table=default.timeout_tst [id=cdcb3e60c1314776843d856365535d49]), partition=HASH (id) PARTITION 4, RANGE (date) PARTITION 20171031 <= VALUES < 20171101 I1031 23:04:22.544366 5869 ts_tablet_manager.cc:862] T bff7dab34ccd4b42842077e82106f6fb P 0f53a0d3ef7e44ebb0365c800752d5bd: Registered tablet (data state: TABLET_DATA_READY) I1031 23:04:22.544734 34571 ts_tablet_manager.cc:723] T bff7dab34ccd4b42842077e82106f6fb P 0f53a0d3ef7e44ebb0365c800752d5bd: Bootstrapping tablet I1031 23:04:22.545177 34571 tablet_bootstrap.cc:382] T bff7dab34ccd4b42842077e82106f6fb P 0f53a0d3ef7e44ebb0365c800752d5bd: Bootstrap starting. I1031 23:04:22.545372 34571 tablet_bootstrap.cc:541] T bff7dab34ccd4b42842077e82106f6fb P 0f53a0d3ef7e44ebb0365c800752d5bd: Time spent opening tablet: real 0.000s user 0.000s sys 0.000s I1031 23:04:22.545522 34571 tablet_bootstrap.cc:484] T bff7dab34ccd4b42842077e82106f6fb P 0f53a0d3ef7e44ebb0365c800752d5bd: No blocks or log segments found. Creating new log. I1031 23:04:22.546749 34571 tablet_bootstrap.cc:382] T bff7dab34ccd4b42842077e82106f6fb P 0f53a0d3ef7e44ebb0365c800752d5bd: No bootstrap required, opened a new log I1031 23:04:22.546778 34571 ts_tablet_manager.cc:728] T bff7dab34ccd4b42842077e82106f6fb P 0f53a0d3ef7e44ebb0365c800752d5bd: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s I1031 23:04:22.547559 34571 raft_consensus.cc:288] T bff7dab34ccd4b42842077e82106f6fb P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f53a0d3ef7e44ebb0365c800752d5bd" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st01.bj" port: 18700 } } peers { permanent_uuid: "23f962e4a1744381ad5fa0d2d8b10241" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st07.bj" port: 18700 } } peers { permanent_uuid: "193181be29284287b25df2ee92938f0b" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st08.bj" port: 18700 } } I1031 23:04:22.547582 34571 raft_consensus.cc:319] T bff7dab34ccd4b42842077e82106f6fb P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1031 23:04:22.547595 34571 raft_consensus.cc:548] T bff7dab34ccd4b42842077e82106f6fb P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f53a0d3ef7e44ebb0365c800752d5bd, State: 1, Role: FOLLOWER I1031 23:04:22.547612 34571 consensus_queue.cc:176] T bff7dab34ccd4b42842077e82106f6fb P 0f53a0d3ef7e44ebb0365c800752d5bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER I1031 23:04:22.569627 5861 ts_tablet_manager.cc:862] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd: Registered tablet (data state: TABLET_DATA_READY) I1031 23:04:22.569893 34576 ts_tablet_manager.cc:723] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd: Bootstrapping tablet I1031 23:04:22.570230 34576 tablet_bootstrap.cc:382] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd: Bootstrap starting. I1031 23:04:22.570358 34576 tablet_bootstrap.cc:541] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd: Time spent opening tablet: real 0.000s user 0.000s sys 0.000s I1031 23:04:22.570468 34576 tablet_bootstrap.cc:484] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd: No blocks or log segments found. Creating new log. I1031 23:04:22.571584 34576 tablet_bootstrap.cc:382] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd: No bootstrap required, opened a new log I1031 23:04:22.571611 34576 ts_tablet_manager.cc:728] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s I1031 23:04:22.572322 34576 raft_consensus.cc:288] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f53a0d3ef7e44ebb0365c800752d5bd" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st01.bj" port: 18700 } } peers { permanent_uuid: "daba39963d4346e1816089fd3a5442b5" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st09.bj" port: 18700 } } peers { permanent_uuid: "23f962e4a1744381ad5fa0d2d8b10241" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st07.bj" port: 18700 } } I1031 23:04:22.572343 34576 raft_consensus.cc:319] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1031 23:04:22.572351 34576 raft_consensus.cc:548] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f53a0d3ef7e44ebb0365c800752d5bd, State: 1, Role: FOLLOWER I1031 23:04:22.572362 34576 consensus_queue.cc:176] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER W1031 23:04:22.585057 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 59 times. I1031 23:04:22.595026 5857 ts_tablet_manager.cc:862] T 56155a2adc24450d89e338cc955bfa18 P 0f53a0d3ef7e44ebb0365c800752d5bd: Registered tablet (data state: TABLET_DATA_READY) I1031 23:04:22.595695 34582 ts_tablet_manager.cc:723] T 56155a2adc24450d89e338cc955bfa18 P 0f53a0d3ef7e44ebb0365c800752d5bd: Bootstrapping tablet I1031 23:04:22.596272 34582 tablet_bootstrap.cc:382] T 56155a2adc24450d89e338cc955bfa18 P 0f53a0d3ef7e44ebb0365c800752d5bd: Bootstrap starting. I1031 23:04:22.596408 34582 tablet_bootstrap.cc:541] T 56155a2adc24450d89e338cc955bfa18 P 0f53a0d3ef7e44ebb0365c800752d5bd: Time spent opening tablet: real 0.000s user 0.000s sys 0.000s I1031 23:04:22.596536 34582 tablet_bootstrap.cc:484] T 56155a2adc24450d89e338cc955bfa18 P 0f53a0d3ef7e44ebb0365c800752d5bd: No blocks or log segments found. Creating new log. I1031 23:04:22.597954 34582 tablet_bootstrap.cc:382] T 56155a2adc24450d89e338cc955bfa18 P 0f53a0d3ef7e44ebb0365c800752d5bd: No bootstrap required, opened a new log I1031 23:04:22.597981 34582 ts_tablet_manager.cc:728] T 56155a2adc24450d89e338cc955bfa18 P 0f53a0d3ef7e44ebb0365c800752d5bd: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s I1031 23:04:22.598786 34582 raft_consensus.cc:288] T 56155a2adc24450d89e338cc955bfa18 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625f49b9554549e6a5c3a92b76e576cf" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st10.bj" port: 18700 } } peers { permanent_uuid: "193181be29284287b25df2ee92938f0b" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st08.bj" port: 18700 } } peers { permanent_uuid: "0f53a0d3ef7e44ebb0365c800752d5bd" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st01.bj" port: 18700 } } I1031 23:04:22.598808 34582 raft_consensus.cc:319] T 56155a2adc24450d89e338cc955bfa18 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1031 23:04:22.598815 34582 raft_consensus.cc:548] T 56155a2adc24450d89e338cc955bfa18 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f53a0d3ef7e44ebb0365c800752d5bd, State: 1, Role: FOLLOWER I1031 23:04:22.598829 34582 consensus_queue.cc:176] T 56155a2adc24450d89e338cc955bfa18 P 0f53a0d3ef7e44ebb0365c800752d5bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER I1031 23:04:22.608784 34585 raft_consensus.cc:411] T 56155a2adc24450d89e338cc955bfa18 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I1031 23:04:22.608814 34585 raft_consensus.cc:2162] T 56155a2adc24450d89e338cc955bfa18 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 2.086s I1031 23:04:22.608862 34585 raft_consensus.cc:435] T 56155a2adc24450d89e338cc955bfa18 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625f49b9554549e6a5c3a92b76e576cf" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st10.bj" port: 18700 } } peers { permanent_uuid: "193181be29284287b25df2ee92938f0b" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st08.bj" port: 18700 } } peers { permanent_uuid: "0f53a0d3ef7e44ebb0365c800752d5bd" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st01.bj" port: 18700 } } I1031 23:04:22.609468 34585 leader_election.cc:216] T 56155a2adc24450d89e338cc955bfa18 P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 625f49b9554549e6a5c3a92b76e576cf I1031 23:04:22.609498 34585 leader_election.cc:216] T 56155a2adc24450d89e338cc955bfa18 P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 193181be29284287b25df2ee92938f0b W1031 23:04:22.609671 5814 leader_election.cc:278] T 56155a2adc24450d89e338cc955bfa18 P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 625f49b9554549e6a5c3a92b76e576cf: Not found: Tablet not found W1031 23:04:22.609702 5815 leader_election.cc:278] T 56155a2adc24450d89e338cc955bfa18 P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 193181be29284287b25df2ee92938f0b: Not found: Tablet not found I1031 23:04:22.609721 5815 leader_election.cc:243] T 56155a2adc24450d89e338cc955bfa18 P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. I1031 23:04:22.609758 34586 raft_consensus.cc:2162] T 56155a2adc24450d89e338cc955bfa18 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.891s I1031 23:04:22.609773 34586 raft_consensus.cc:1908] T 56155a2adc24450d89e338cc955bfa18 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority I1031 23:04:22.620036 5866 ts_tablet_manager.cc:862] T 117cd74bcdab4942901acb0005cdd702 P 0f53a0d3ef7e44ebb0365c800752d5bd: Registered tablet (data state: TABLET_DATA_READY) I1031 23:04:22.620532 34587 ts_tablet_manager.cc:723] T 117cd74bcdab4942901acb0005cdd702 P 0f53a0d3ef7e44ebb0365c800752d5bd: Bootstrapping tablet I1031 23:04:22.620920 34587 tablet_bootstrap.cc:382] T 117cd74bcdab4942901acb0005cdd702 P 0f53a0d3ef7e44ebb0365c800752d5bd: Bootstrap starting. I1031 23:04:22.621042 34587 tablet_bootstrap.cc:541] T 117cd74bcdab4942901acb0005cdd702 P 0f53a0d3ef7e44ebb0365c800752d5bd: Time spent opening tablet: real 0.000s user 0.000s sys 0.000s I1031 23:04:22.621136 34587 tablet_bootstrap.cc:484] T 117cd74bcdab4942901acb0005cdd702 P 0f53a0d3ef7e44ebb0365c800752d5bd: No blocks or log segments found. Creating new log. I1031 23:04:22.622375 34587 tablet_bootstrap.cc:382] T 117cd74bcdab4942901acb0005cdd702 P 0f53a0d3ef7e44ebb0365c800752d5bd: No bootstrap required, opened a new log I1031 23:04:22.622402 34587 ts_tablet_manager.cc:728] T 117cd74bcdab4942901acb0005cdd702 P 0f53a0d3ef7e44ebb0365c800752d5bd: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s I1031 23:04:22.622836 34587 raft_consensus.cc:288] T 117cd74bcdab4942901acb0005cdd702 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625f49b9554549e6a5c3a92b76e576cf" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st10.bj" port: 18700 } } peers { permanent_uuid: "0a8bbcbbb5a6459aa5258464755db3bb" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st02.bj" port: 18700 } } peers { permanent_uuid: "0f53a0d3ef7e44ebb0365c800752d5bd" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st01.bj" port: 18700 } } I1031 23:04:22.622856 34587 raft_consensus.cc:319] T 117cd74bcdab4942901acb0005cdd702 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1031 23:04:22.622867 34587 raft_consensus.cc:548] T 117cd74bcdab4942901acb0005cdd702 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f53a0d3ef7e44ebb0365c800752d5bd, State: 1, Role: FOLLOWER I1031 23:04:22.622889 34587 consensus_queue.cc:176] T 117cd74bcdab4942901acb0005cdd702 P 0f53a0d3ef7e44ebb0365c800752d5bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER I1031 23:04:22.768848 34590 raft_consensus.cc:411] T 117cd74bcdab4942901acb0005cdd702 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I1031 23:04:22.768903 34590 raft_consensus.cc:2162] T 117cd74bcdab4942901acb0005cdd702 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.573s I1031 23:04:22.768957 34590 raft_consensus.cc:435] T 117cd74bcdab4942901acb0005cdd702 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625f49b9554549e6a5c3a92b76e576cf" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st10.bj" port: 18700 } } peers { permanent_uuid: "0a8bbcbbb5a6459aa5258464755db3bb" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st02.bj" port: 18700 } } peers { permanent_uuid: "0f53a0d3ef7e44ebb0365c800752d5bd" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st01.bj" port: 18700 } } I1031 23:04:22.769657 34590 leader_election.cc:216] T 117cd74bcdab4942901acb0005cdd702 P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 625f49b9554549e6a5c3a92b76e576cf I1031 23:04:22.769700 34590 leader_election.cc:216] T 117cd74bcdab4942901acb0005cdd702 P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 0a8bbcbbb5a6459aa5258464755db3bb W1031 23:04:22.769970 5816 leader_election.cc:278] T 117cd74bcdab4942901acb0005cdd702 P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0a8bbcbbb5a6459aa5258464755db3bb: Not found: Tablet not found W1031 23:04:22.770005 5814 leader_election.cc:278] T 117cd74bcdab4942901acb0005cdd702 P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 625f49b9554549e6a5c3a92b76e576cf: Illegal state: Tablet not RUNNING: NOT_STARTED I1031 23:04:22.770022 5814 leader_election.cc:243] T 117cd74bcdab4942901acb0005cdd702 P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. I1031 23:04:22.770087 34591 raft_consensus.cc:2162] T 117cd74bcdab4942901acb0005cdd702 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.825s I1031 23:04:22.770102 34591 raft_consensus.cc:1908] T 117cd74bcdab4942901acb0005cdd702 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority W1031 23:04:22.837774 5816 consensus_peers.cc:357] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 0a8bbcbbb5a6459aa5258464755db3bb (c3-kudu-tst-st02.bj:18700): Couldn't send request to peer 0a8bbcbbb5a6459aa5258464755db3bb for tablet 648eb376255a4dd4bb5818262efcc813. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 60 times. W1031 23:04:23.085237 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 60 times. I1031 23:04:23.159570 34574 raft_consensus.cc:411] T bff7dab34ccd4b42842077e82106f6fb P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I1031 23:04:23.159606 34574 raft_consensus.cc:2162] T bff7dab34ccd4b42842077e82106f6fb P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.823s I1031 23:04:23.159667 34574 raft_consensus.cc:435] T bff7dab34ccd4b42842077e82106f6fb P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f53a0d3ef7e44ebb0365c800752d5bd" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st01.bj" port: 18700 } } peers { permanent_uuid: "23f962e4a1744381ad5fa0d2d8b10241" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st07.bj" port: 18700 } } peers { permanent_uuid: "193181be29284287b25df2ee92938f0b" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st08.bj" port: 18700 } } I1031 23:04:23.160451 34574 leader_election.cc:216] T bff7dab34ccd4b42842077e82106f6fb P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 23f962e4a1744381ad5fa0d2d8b10241 I1031 23:04:23.160501 34574 leader_election.cc:216] T bff7dab34ccd4b42842077e82106f6fb P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 193181be29284287b25df2ee92938f0b W1031 23:04:23.160756 5816 leader_election.cc:278] T bff7dab34ccd4b42842077e82106f6fb P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 23f962e4a1744381ad5fa0d2d8b10241: Illegal state: Tablet not RUNNING: NOT_STARTED W1031 23:04:23.160807 5815 leader_election.cc:278] T bff7dab34ccd4b42842077e82106f6fb P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 193181be29284287b25df2ee92938f0b: Illegal state: Tablet not RUNNING: NOT_STARTED I1031 23:04:23.160828 5815 leader_election.cc:243] T bff7dab34ccd4b42842077e82106f6fb P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. I1031 23:04:23.160876 34575 raft_consensus.cc:2162] T bff7dab34ccd4b42842077e82106f6fb P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 2.068s I1031 23:04:23.160905 34575 raft_consensus.cc:1908] T bff7dab34ccd4b42842077e82106f6fb P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority I1031 23:04:23.248337 34579 raft_consensus.cc:411] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I1031 23:04:23.248380 34579 raft_consensus.cc:2162] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.954s I1031 23:04:23.248421 34579 raft_consensus.cc:435] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f53a0d3ef7e44ebb0365c800752d5bd" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st01.bj" port: 18700 } } peers { permanent_uuid: "daba39963d4346e1816089fd3a5442b5" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st09.bj" port: 18700 } } peers { permanent_uuid: "23f962e4a1744381ad5fa0d2d8b10241" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st07.bj" port: 18700 } } I1031 23:04:23.249158 34579 leader_election.cc:216] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer daba39963d4346e1816089fd3a5442b5 I1031 23:04:23.249207 34579 leader_election.cc:216] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 23f962e4a1744381ad5fa0d2d8b10241 W1031 23:04:23.249544 5816 leader_election.cc:278] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 23f962e4a1744381ad5fa0d2d8b10241: Illegal state: Tablet not RUNNING: NOT_STARTED I1031 23:04:23.249586 5813 leader_election.cc:348] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Vote granted by peer daba39963d4346e1816089fd3a5442b5 I1031 23:04:23.249600 5813 leader_election.cc:243] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. I1031 23:04:23.249678 34581 raft_consensus.cc:2162] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.662s I1031 23:04:23.249699 34581 raft_consensus.cc:1971] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Leader pre-election won for term 1 I1031 23:04:23.249707 34581 raft_consensus.cc:411] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I1031 23:04:23.249729 34581 raft_consensus.cc:2162] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.647s I1031 23:04:23.249737 34581 raft_consensus.cc:2206] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Advancing to term 1 I1031 23:04:23.250430 34581 raft_consensus.cc:435] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f53a0d3ef7e44ebb0365c800752d5bd" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st01.bj" port: 18700 } } peers { permanent_uuid: "daba39963d4346e1816089fd3a5442b5" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st09.bj" port: 18700 } } peers { permanent_uuid: "23f962e4a1744381ad5fa0d2d8b10241" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st07.bj" port: 18700 } } I1031 23:04:23.250918 34581 leader_election.cc:216] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 election: Requesting vote from peer daba39963d4346e1816089fd3a5442b5 I1031 23:04:23.250957 34581 leader_election.cc:216] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 election: Requesting vote from peer 23f962e4a1744381ad5fa0d2d8b10241 W1031 23:04:23.251226 5816 leader_election.cc:278] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 23f962e4a1744381ad5fa0d2d8b10241: Illegal state: Tablet not RUNNING: NOT_STARTED I1031 23:04:23.251823 5813 leader_election.cc:348] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 election: Vote granted by peer daba39963d4346e1816089fd3a5442b5 I1031 23:04:23.251837 5813 leader_election.cc:243] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I1031 23:04:23.251873 34581 raft_consensus.cc:2162] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 2.043s I1031 23:04:23.251888 34581 raft_consensus.cc:1971] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 1 FOLLOWER]: Leader election won for term 1 I1031 23:04:23.252166 34581 raft_consensus.cc:518] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 1 LEADER]: Becoming Leader. State: Replica: 0f53a0d3ef7e44ebb0365c800752d5bd, State: 1, Role: LEADER I1031 23:04:23.252241 34581 consensus_queue.cc:158] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Current term: 1, Majority size: 2, State: 1, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f53a0d3ef7e44ebb0365c800752d5bd" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st01.bj" port: 18700 } } peers { permanent_uuid: "daba39963d4346e1816089fd3a5442b5" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st09.bj" port: 18700 } } peers { permanent_uuid: "23f962e4a1744381ad5fa0d2d8b10241" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st07.bj" port: 18700 } } W1031 23:04:23.337859 5816 consensus_peers.cc:357] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 0a8bbcbbb5a6459aa5258464755db3bb (c3-kudu-tst-st02.bj:18700): Couldn't send request to peer 0a8bbcbbb5a6459aa5258464755db3bb for tablet 648eb376255a4dd4bb5818262efcc813. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 61 times. W1031 23:04:23.585410 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 61 times. I1031 23:04:23.753021 34581 consensus_queue.cc:695] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd [LEADER]: Connected to new peer: Peer: daba39963d4346e1816089fd3a5442b5, Is new: false, Last received: 0.0, Next index: 1, Last known committed idx: 0, Last exchange result: ERROR, Needs tablet copy: false W1031 23:04:23.753427 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 1 times. W1031 23:04:23.837947 5816 consensus_peers.cc:357] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 0a8bbcbbb5a6459aa5258464755db3bb (c3-kudu-tst-st02.bj:18700): Couldn't send request to peer 0a8bbcbbb5a6459aa5258464755db3bb for tablet 648eb376255a4dd4bb5818262efcc813. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 62 times. W1031 23:04:24.085512 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 62 times. W1031 23:04:24.253568 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 2 times. W1031 23:04:24.338033 5816 consensus_peers.cc:357] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 0a8bbcbbb5a6459aa5258464755db3bb (c3-kudu-tst-st02.bj:18700): Couldn't send request to peer 0a8bbcbbb5a6459aa5258464755db3bb for tablet 648eb376255a4dd4bb5818262efcc813. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 63 times. W1031 23:04:24.570755 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 3 times. W1031 23:04:24.585475 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 63 times. W1031 23:04:24.838049 5816 consensus_peers.cc:357] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 0a8bbcbbb5a6459aa5258464755db3bb (c3-kudu-tst-st02.bj:18700): Couldn't send request to peer 0a8bbcbbb5a6459aa5258464755db3bb for tablet 648eb376255a4dd4bb5818262efcc813. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 64 times. W1031 23:04:25.070783 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 4 times. W1031 23:04:25.085602 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 64 times. W1031 23:04:25.338165 5816 consensus_peers.cc:357] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 0a8bbcbbb5a6459aa5258464755db3bb (c3-kudu-tst-st02.bj:18700): Couldn't send request to peer 0a8bbcbbb5a6459aa5258464755db3bb for tablet 648eb376255a4dd4bb5818262efcc813. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 65 times. W1031 23:04:25.570884 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 5 times. W1031 23:04:25.585737 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 65 times. W1031 23:04:25.838186 5816 consensus_peers.cc:357] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 0a8bbcbbb5a6459aa5258464755db3bb (c3-kudu-tst-st02.bj:18700): Couldn't send request to peer 0a8bbcbbb5a6459aa5258464755db3bb for tablet 648eb376255a4dd4bb5818262efcc813. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 66 times. I1031 23:04:26.014925 34585 raft_consensus.cc:411] T 56155a2adc24450d89e338cc955bfa18 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I1031 23:04:26.014966 34585 raft_consensus.cc:2162] T 56155a2adc24450d89e338cc955bfa18 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.968s I1031 23:04:26.015029 34585 raft_consensus.cc:435] T 56155a2adc24450d89e338cc955bfa18 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625f49b9554549e6a5c3a92b76e576cf" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st10.bj" port: 18700 } } peers { permanent_uuid: "193181be29284287b25df2ee92938f0b" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st08.bj" port: 18700 } } peers { permanent_uuid: "0f53a0d3ef7e44ebb0365c800752d5bd" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st01.bj" port: 18700 } } I1031 23:04:26.015581 34585 leader_election.cc:216] T 56155a2adc24450d89e338cc955bfa18 P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 625f49b9554549e6a5c3a92b76e576cf I1031 23:04:26.015609 34585 leader_election.cc:216] T 56155a2adc24450d89e338cc955bfa18 P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 193181be29284287b25df2ee92938f0b W1031 23:04:26.015888 5814 leader_election.cc:278] T 56155a2adc24450d89e338cc955bfa18 P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 625f49b9554549e6a5c3a92b76e576cf: Illegal state: Tablet not RUNNING: NOT_STARTED W1031 23:04:26.015930 5815 leader_election.cc:278] T 56155a2adc24450d89e338cc955bfa18 P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 193181be29284287b25df2ee92938f0b: Illegal state: Tablet not RUNNING: NOT_STARTED I1031 23:04:26.015956 5815 leader_election.cc:243] T 56155a2adc24450d89e338cc955bfa18 P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. I1031 23:04:26.016167 34616 raft_consensus.cc:2162] T 56155a2adc24450d89e338cc955bfa18 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 2.210s I1031 23:04:26.016185 34616 raft_consensus.cc:1908] T 56155a2adc24450d89e338cc955bfa18 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority W1031 23:04:26.070932 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 6 times. W1031 23:04:26.085786 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 66 times. W1031 23:04:26.338297 5816 consensus_peers.cc:357] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 0a8bbcbbb5a6459aa5258464755db3bb (c3-kudu-tst-st02.bj:18700): Couldn't send request to peer 0a8bbcbbb5a6459aa5258464755db3bb for tablet 648eb376255a4dd4bb5818262efcc813. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 67 times. I1031 23:04:26.533145 34590 raft_consensus.cc:411] T 117cd74bcdab4942901acb0005cdd702 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I1031 23:04:26.533215 34590 raft_consensus.cc:2162] T 117cd74bcdab4942901acb0005cdd702 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 2.181s I1031 23:04:26.533291 34590 raft_consensus.cc:435] T 117cd74bcdab4942901acb0005cdd702 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625f49b9554549e6a5c3a92b76e576cf" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st10.bj" port: 18700 } } peers { permanent_uuid: "0a8bbcbbb5a6459aa5258464755db3bb" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st02.bj" port: 18700 } } peers { permanent_uuid: "0f53a0d3ef7e44ebb0365c800752d5bd" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st01.bj" port: 18700 } } I1031 23:04:26.533834 34590 leader_election.cc:216] T 117cd74bcdab4942901acb0005cdd702 P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 625f49b9554549e6a5c3a92b76e576cf I1031 23:04:26.533859 34590 leader_election.cc:216] T 117cd74bcdab4942901acb0005cdd702 P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 0a8bbcbbb5a6459aa5258464755db3bb W1031 23:04:26.534122 5816 leader_election.cc:278] T 117cd74bcdab4942901acb0005cdd702 P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0a8bbcbbb5a6459aa5258464755db3bb: Illegal state: Tablet not RUNNING: NOT_STARTED W1031 23:04:26.534159 5814 leader_election.cc:278] T 117cd74bcdab4942901acb0005cdd702 P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 625f49b9554549e6a5c3a92b76e576cf: Illegal state: Tablet not RUNNING: NOT_STARTED I1031 23:04:26.534206 5814 leader_election.cc:243] T 117cd74bcdab4942901acb0005cdd702 P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. I1031 23:04:26.534459 34621 raft_consensus.cc:2162] T 117cd74bcdab4942901acb0005cdd702 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.898s I1031 23:04:26.534482 34621 raft_consensus.cc:1908] T 117cd74bcdab4942901acb0005cdd702 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority W1031 23:04:26.571034 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 7 times. W1031 23:04:26.585876 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 67 times. W1031 23:04:26.838333 5816 consensus_peers.cc:357] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 0a8bbcbbb5a6459aa5258464755db3bb (c3-kudu-tst-st02.bj:18700): Couldn't send request to peer 0a8bbcbbb5a6459aa5258464755db3bb for tablet 648eb376255a4dd4bb5818262efcc813. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 68 times. W1031 23:04:27.070977 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 8 times. I1031 23:04:27.074112 34574 raft_consensus.cc:411] T bff7dab34ccd4b42842077e82106f6fb P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I1031 23:04:27.074144 34574 raft_consensus.cc:2162] T bff7dab34ccd4b42842077e82106f6fb P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 3.076s I1031 23:04:27.074209 34574 raft_consensus.cc:435] T bff7dab34ccd4b42842077e82106f6fb P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f53a0d3ef7e44ebb0365c800752d5bd" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st01.bj" port: 18700 } } peers { permanent_uuid: "23f962e4a1744381ad5fa0d2d8b10241" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st07.bj" port: 18700 } } peers { permanent_uuid: "193181be29284287b25df2ee92938f0b" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st08.bj" port: 18700 } } I1031 23:04:27.074777 34574 leader_election.cc:216] T bff7dab34ccd4b42842077e82106f6fb P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 23f962e4a1744381ad5fa0d2d8b10241 I1031 23:04:27.074807 34574 leader_election.cc:216] T bff7dab34ccd4b42842077e82106f6fb P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 193181be29284287b25df2ee92938f0b W1031 23:04:27.075042 5816 leader_election.cc:278] T bff7dab34ccd4b42842077e82106f6fb P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 23f962e4a1744381ad5fa0d2d8b10241: Illegal state: Tablet not RUNNING: NOT_STARTED W1031 23:04:27.075166 5815 leader_election.cc:278] T bff7dab34ccd4b42842077e82106f6fb P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 193181be29284287b25df2ee92938f0b: Illegal state: Tablet not RUNNING: NOT_STARTED I1031 23:04:27.075198 5815 leader_election.cc:243] T bff7dab34ccd4b42842077e82106f6fb P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. I1031 23:04:27.075621 34622 raft_consensus.cc:2162] T bff7dab34ccd4b42842077e82106f6fb P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 3.163s I1031 23:04:27.075644 34622 raft_consensus.cc:1908] T bff7dab34ccd4b42842077e82106f6fb P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority W1031 23:04:27.085965 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 68 times. W1031 23:04:27.338415 5816 consensus_peers.cc:357] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 0a8bbcbbb5a6459aa5258464755db3bb (c3-kudu-tst-st02.bj:18700): Couldn't send request to peer 0a8bbcbbb5a6459aa5258464755db3bb for tablet 648eb376255a4dd4bb5818262efcc813. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 69 times. W1031 23:04:27.571089 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 9 times. W1031 23:04:27.585970 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 69 times. W1031 23:04:27.838508 5816 consensus_peers.cc:357] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 0a8bbcbbb5a6459aa5258464755db3bb (c3-kudu-tst-st02.bj:18700): Couldn't send request to peer 0a8bbcbbb5a6459aa5258464755db3bb for tablet 648eb376255a4dd4bb5818262efcc813. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 70 times. W1031 23:04:28.071233 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 10 times. W1031 23:04:28.086172 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 70 times. W1031 23:04:28.338634 5816 consensus_peers.cc:357] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 0a8bbcbbb5a6459aa5258464755db3bb (c3-kudu-tst-st02.bj:18700): Couldn't send request to peer 0a8bbcbbb5a6459aa5258464755db3bb for tablet 648eb376255a4dd4bb5818262efcc813. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 71 times. W1031 23:04:28.571310 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 11 times. W1031 23:04:28.586243 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 71 times. W1031 23:04:28.838755 5816 consensus_peers.cc:357] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 0a8bbcbbb5a6459aa5258464755db3bb (c3-kudu-tst-st02.bj:18700): Couldn't send request to peer 0a8bbcbbb5a6459aa5258464755db3bb for tablet 648eb376255a4dd4bb5818262efcc813. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 72 times. W1031 23:04:29.071347 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 12 times. W1031 23:04:29.086284 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 72 times. W1031 23:04:29.338752 5816 consensus_peers.cc:357] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 0a8bbcbbb5a6459aa5258464755db3bb (c3-kudu-tst-st02.bj:18700): Couldn't send request to peer 0a8bbcbbb5a6459aa5258464755db3bb for tablet 648eb376255a4dd4bb5818262efcc813. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 73 times. W1031 23:04:29.571486 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 13 times. W1031 23:04:29.586380 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 73 times. W1031 23:04:29.838789 5816 consensus_peers.cc:357] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 0a8bbcbbb5a6459aa5258464755db3bb (c3-kudu-tst-st02.bj:18700): Couldn't send request to peer 0a8bbcbbb5a6459aa5258464755db3bb for tablet 648eb376255a4dd4bb5818262efcc813. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 74 times. W1031 23:04:30.071432 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 14 times. W1031 23:04:30.086422 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 74 times. I1031 23:04:30.310490 34590 raft_consensus.cc:411] T 117cd74bcdab4942901acb0005cdd702 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I1031 23:04:30.310533 34590 raft_consensus.cc:2162] T 117cd74bcdab4942901acb0005cdd702 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 3.090s I1031 23:04:30.310591 34590 raft_consensus.cc:435] T 117cd74bcdab4942901acb0005cdd702 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625f49b9554549e6a5c3a92b76e576cf" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st10.bj" port: 18700 } } peers { permanent_uuid: "0a8bbcbbb5a6459aa5258464755db3bb" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st02.bj" port: 18700 } } peers { permanent_uuid: "0f53a0d3ef7e44ebb0365c800752d5bd" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st01.bj" port: 18700 } } I1031 23:04:30.311161 34590 leader_election.cc:216] T 117cd74bcdab4942901acb0005cdd702 P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 625f49b9554549e6a5c3a92b76e576cf I1031 23:04:30.311184 34590 leader_election.cc:216] T 117cd74bcdab4942901acb0005cdd702 P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 0a8bbcbbb5a6459aa5258464755db3bb W1031 23:04:30.311450 5816 leader_election.cc:278] T 117cd74bcdab4942901acb0005cdd702 P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0a8bbcbbb5a6459aa5258464755db3bb: Illegal state: Tablet not RUNNING: NOT_STARTED W1031 23:04:30.311501 5814 leader_election.cc:278] T 117cd74bcdab4942901acb0005cdd702 P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 625f49b9554549e6a5c3a92b76e576cf: Illegal state: Tablet not RUNNING: NOT_STARTED I1031 23:04:30.311532 5814 leader_election.cc:243] T 117cd74bcdab4942901acb0005cdd702 P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. I1031 23:04:30.312292 34632 raft_consensus.cc:2162] T 117cd74bcdab4942901acb0005cdd702 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 4.077s I1031 23:04:30.312309 34632 raft_consensus.cc:1908] T 117cd74bcdab4942901acb0005cdd702 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority W1031 23:04:30.338901 5816 consensus_peers.cc:357] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 0a8bbcbbb5a6459aa5258464755db3bb (c3-kudu-tst-st02.bj:18700): Couldn't send request to peer 0a8bbcbbb5a6459aa5258464755db3bb for tablet 648eb376255a4dd4bb5818262efcc813. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 75 times. I1031 23:04:30.378440 34585 raft_consensus.cc:411] T 56155a2adc24450d89e338cc955bfa18 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I1031 23:04:30.378466 34585 raft_consensus.cc:2162] T 56155a2adc24450d89e338cc955bfa18 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.636s I1031 23:04:30.378504 34585 raft_consensus.cc:435] T 56155a2adc24450d89e338cc955bfa18 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625f49b9554549e6a5c3a92b76e576cf" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st10.bj" port: 18700 } } peers { permanent_uuid: "193181be29284287b25df2ee92938f0b" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st08.bj" port: 18700 } } peers { permanent_uuid: "0f53a0d3ef7e44ebb0365c800752d5bd" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st01.bj" port: 18700 } } I1031 23:04:30.379094 34585 leader_election.cc:216] T 56155a2adc24450d89e338cc955bfa18 P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 625f49b9554549e6a5c3a92b76e576cf I1031 23:04:30.379123 34585 leader_election.cc:216] T 56155a2adc24450d89e338cc955bfa18 P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 193181be29284287b25df2ee92938f0b W1031 23:04:30.379427 5814 leader_election.cc:278] T 56155a2adc24450d89e338cc955bfa18 P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 625f49b9554549e6a5c3a92b76e576cf: Illegal state: Tablet not RUNNING: NOT_STARTED W1031 23:04:30.379473 5815 leader_election.cc:278] T 56155a2adc24450d89e338cc955bfa18 P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 193181be29284287b25df2ee92938f0b: Illegal state: Tablet not RUNNING: NOT_STARTED I1031 23:04:30.379521 5815 leader_election.cc:243] T 56155a2adc24450d89e338cc955bfa18 P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. I1031 23:04:30.379835 34633 raft_consensus.cc:2162] T 56155a2adc24450d89e338cc955bfa18 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 2.299s I1031 23:04:30.379868 34633 raft_consensus.cc:1908] T 56155a2adc24450d89e338cc955bfa18 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority W1031 23:04:30.571595 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 15 times. W1031 23:04:30.586604 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 75 times. W1031 23:04:30.838937 5816 consensus_peers.cc:357] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 0a8bbcbbb5a6459aa5258464755db3bb (c3-kudu-tst-st02.bj:18700): Couldn't send request to peer 0a8bbcbbb5a6459aa5258464755db3bb for tablet 648eb376255a4dd4bb5818262efcc813. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 76 times. W1031 23:04:31.071619 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 16 times. W1031 23:04:31.086668 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 76 times. W1031 23:04:31.339035 5816 consensus_peers.cc:357] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 0a8bbcbbb5a6459aa5258464755db3bb (c3-kudu-tst-st02.bj:18700): Couldn't send request to peer 0a8bbcbbb5a6459aa5258464755db3bb for tablet 648eb376255a4dd4bb5818262efcc813. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 77 times. W1031 23:04:31.571739 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 17 times. W1031 23:04:31.586702 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 77 times. W1031 23:04:31.839104 5816 consensus_peers.cc:357] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 0a8bbcbbb5a6459aa5258464755db3bb (c3-kudu-tst-st02.bj:18700): Couldn't send request to peer 0a8bbcbbb5a6459aa5258464755db3bb for tablet 648eb376255a4dd4bb5818262efcc813. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 78 times. I1031 23:04:31.995713 34574 raft_consensus.cc:411] T bff7dab34ccd4b42842077e82106f6fb P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I1031 23:04:31.995754 34574 raft_consensus.cc:2162] T bff7dab34ccd4b42842077e82106f6fb P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 2.196s I1031 23:04:31.995805 34574 raft_consensus.cc:435] T bff7dab34ccd4b42842077e82106f6fb P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f53a0d3ef7e44ebb0365c800752d5bd" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st01.bj" port: 18700 } } peers { permanent_uuid: "23f962e4a1744381ad5fa0d2d8b10241" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st07.bj" port: 18700 } } peers { permanent_uuid: "193181be29284287b25df2ee92938f0b" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st08.bj" port: 18700 } } I1031 23:04:31.996573 34574 leader_election.cc:216] T bff7dab34ccd4b42842077e82106f6fb P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 23f962e4a1744381ad5fa0d2d8b10241 I1031 23:04:31.996604 34574 leader_election.cc:216] T bff7dab34ccd4b42842077e82106f6fb P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 193181be29284287b25df2ee92938f0b W1031 23:04:31.996866 5816 leader_election.cc:278] T bff7dab34ccd4b42842077e82106f6fb P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 23f962e4a1744381ad5fa0d2d8b10241: Illegal state: Tablet not RUNNING: NOT_STARTED W1031 23:04:31.996937 5815 leader_election.cc:278] T bff7dab34ccd4b42842077e82106f6fb P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 193181be29284287b25df2ee92938f0b: Illegal state: Tablet not RUNNING: NOT_STARTED I1031 23:04:31.996973 5815 leader_election.cc:243] T bff7dab34ccd4b42842077e82106f6fb P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. I1031 23:04:31.997228 34636 raft_consensus.cc:2162] T bff7dab34ccd4b42842077e82106f6fb P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 4.366s I1031 23:04:31.997246 34636 raft_consensus.cc:1908] T bff7dab34ccd4b42842077e82106f6fb P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority W1031 23:04:32.071761 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 18 times. W1031 23:04:32.086838 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 78 times. W1031 23:04:32.339249 5816 consensus_peers.cc:357] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 0a8bbcbbb5a6459aa5258464755db3bb (c3-kudu-tst-st02.bj:18700): Couldn't send request to peer 0a8bbcbbb5a6459aa5258464755db3bb for tablet 648eb376255a4dd4bb5818262efcc813. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 79 times. W1031 23:04:32.571892 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 19 times. W1031 23:04:32.586823 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 79 times. W1031 23:04:32.839249 5816 consensus_peers.cc:357] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 0a8bbcbbb5a6459aa5258464755db3bb (c3-kudu-tst-st02.bj:18700): Couldn't send request to peer 0a8bbcbbb5a6459aa5258464755db3bb for tablet 648eb376255a4dd4bb5818262efcc813. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 80 times. W1031 23:04:33.071964 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 20 times. W1031 23:04:33.087081 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 80 times. W1031 23:04:33.339429 5816 consensus_peers.cc:357] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 0a8bbcbbb5a6459aa5258464755db3bb (c3-kudu-tst-st02.bj:18700): Couldn't send request to peer 0a8bbcbbb5a6459aa5258464755db3bb for tablet 648eb376255a4dd4bb5818262efcc813. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 81 times. W1031 23:04:33.572058 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 21 times. W1031 23:04:33.587113 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 81 times. W1031 23:04:33.839388 5816 consensus_peers.cc:357] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 0a8bbcbbb5a6459aa5258464755db3bb (c3-kudu-tst-st02.bj:18700): Couldn't send request to peer 0a8bbcbbb5a6459aa5258464755db3bb for tablet 648eb376255a4dd4bb5818262efcc813. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 82 times. W1031 23:04:34.072088 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 22 times. W1031 23:04:34.087226 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 82 times. I1031 23:04:34.295115 34585 raft_consensus.cc:411] T 56155a2adc24450d89e338cc955bfa18 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I1031 23:04:34.295167 34585 raft_consensus.cc:2162] T 56155a2adc24450d89e338cc955bfa18 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 3.715s I1031 23:04:34.295222 34585 raft_consensus.cc:435] T 56155a2adc24450d89e338cc955bfa18 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625f49b9554549e6a5c3a92b76e576cf" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st10.bj" port: 18700 } } peers { permanent_uuid: "193181be29284287b25df2ee92938f0b" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st08.bj" port: 18700 } } peers { permanent_uuid: "0f53a0d3ef7e44ebb0365c800752d5bd" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st01.bj" port: 18700 } } I1031 23:04:34.295783 34585 leader_election.cc:216] T 56155a2adc24450d89e338cc955bfa18 P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 625f49b9554549e6a5c3a92b76e576cf I1031 23:04:34.295814 34585 leader_election.cc:216] T 56155a2adc24450d89e338cc955bfa18 P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 193181be29284287b25df2ee92938f0b W1031 23:04:34.296072 5815 leader_election.cc:278] T 56155a2adc24450d89e338cc955bfa18 P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 193181be29284287b25df2ee92938f0b: Illegal state: Tablet not RUNNING: NOT_STARTED W1031 23:04:34.296131 5814 leader_election.cc:278] T 56155a2adc24450d89e338cc955bfa18 P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 625f49b9554549e6a5c3a92b76e576cf: Illegal state: Tablet not RUNNING: NOT_STARTED I1031 23:04:34.296154 5814 leader_election.cc:243] T 56155a2adc24450d89e338cc955bfa18 P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. I1031 23:04:34.296350 34643 raft_consensus.cc:2162] T 56155a2adc24450d89e338cc955bfa18 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 6.432s I1031 23:04:34.296373 34643 raft_consensus.cc:1908] T 56155a2adc24450d89e338cc955bfa18 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority W1031 23:04:34.339470 5816 consensus_peers.cc:357] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 0a8bbcbbb5a6459aa5258464755db3bb (c3-kudu-tst-st02.bj:18700): Couldn't send request to peer 0a8bbcbbb5a6459aa5258464755db3bb for tablet 648eb376255a4dd4bb5818262efcc813. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 83 times. W1031 23:04:34.572145 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 23 times. W1031 23:04:34.587261 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 83 times. W1031 23:04:34.839596 5816 consensus_peers.cc:357] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 0a8bbcbbb5a6459aa5258464755db3bb (c3-kudu-tst-st02.bj:18700): Couldn't send request to peer 0a8bbcbbb5a6459aa5258464755db3bb for tablet 648eb376255a4dd4bb5818262efcc813. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 84 times. W1031 23:04:35.072203 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 24 times. W1031 23:04:35.087361 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 84 times. W1031 23:04:35.339689 5816 consensus_peers.cc:357] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 0a8bbcbbb5a6459aa5258464755db3bb (c3-kudu-tst-st02.bj:18700): Couldn't send request to peer 0a8bbcbbb5a6459aa5258464755db3bb for tablet 648eb376255a4dd4bb5818262efcc813. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 85 times. W1031 23:04:35.572404 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 25 times. W1031 23:04:35.587483 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 85 times. W1031 23:04:35.839725 5816 consensus_peers.cc:357] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 0a8bbcbbb5a6459aa5258464755db3bb (c3-kudu-tst-st02.bj:18700): Couldn't send request to peer 0a8bbcbbb5a6459aa5258464755db3bb for tablet 648eb376255a4dd4bb5818262efcc813. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 86 times. W1031 23:04:36.072399 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 26 times. W1031 23:04:36.087520 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 86 times. I1031 23:04:36.129099 34590 raft_consensus.cc:411] T 117cd74bcdab4942901acb0005cdd702 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I1031 23:04:36.129132 34590 raft_consensus.cc:2162] T 117cd74bcdab4942901acb0005cdd702 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 7.088s I1031 23:04:36.129189 34590 raft_consensus.cc:435] T 117cd74bcdab4942901acb0005cdd702 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625f49b9554549e6a5c3a92b76e576cf" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st10.bj" port: 18700 } } peers { permanent_uuid: "0a8bbcbbb5a6459aa5258464755db3bb" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st02.bj" port: 18700 } } peers { permanent_uuid: "0f53a0d3ef7e44ebb0365c800752d5bd" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st01.bj" port: 18700 } } I1031 23:04:36.129707 34590 leader_election.cc:216] T 117cd74bcdab4942901acb0005cdd702 P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 625f49b9554549e6a5c3a92b76e576cf I1031 23:04:36.129736 34590 leader_election.cc:216] T 117cd74bcdab4942901acb0005cdd702 P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 0a8bbcbbb5a6459aa5258464755db3bb W1031 23:04:36.130007 5816 leader_election.cc:278] T 117cd74bcdab4942901acb0005cdd702 P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0a8bbcbbb5a6459aa5258464755db3bb: Illegal state: Tablet not RUNNING: NOT_STARTED W1031 23:04:36.130046 5814 leader_election.cc:278] T 117cd74bcdab4942901acb0005cdd702 P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 625f49b9554549e6a5c3a92b76e576cf: Illegal state: Tablet not RUNNING: NOT_STARTED I1031 23:04:36.130069 5814 leader_election.cc:243] T 117cd74bcdab4942901acb0005cdd702 P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. I1031 23:04:36.130254 34676 raft_consensus.cc:2162] T 117cd74bcdab4942901acb0005cdd702 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 7.470s I1031 23:04:36.130270 34676 raft_consensus.cc:1908] T 117cd74bcdab4942901acb0005cdd702 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority W1031 23:04:36.339772 5816 consensus_peers.cc:357] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 0a8bbcbbb5a6459aa5258464755db3bb (c3-kudu-tst-st02.bj:18700): Couldn't send request to peer 0a8bbcbbb5a6459aa5258464755db3bb for tablet 648eb376255a4dd4bb5818262efcc813. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 87 times. W1031 23:04:36.572371 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 27 times. W1031 23:04:36.587611 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 87 times. W1031 23:04:36.839862 5816 consensus_peers.cc:357] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 0a8bbcbbb5a6459aa5258464755db3bb (c3-kudu-tst-st02.bj:18700): Couldn't send request to peer 0a8bbcbbb5a6459aa5258464755db3bb for tablet 648eb376255a4dd4bb5818262efcc813. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 88 times. W1031 23:04:37.072535 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 28 times. W1031 23:04:37.087680 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 88 times. I1031 23:04:37.300109 34067 raft_consensus.cc:411] T 6a813ca355f749b6b31a773aa0f2ac7a P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I1031 23:04:37.300173 34067 raft_consensus.cc:2162] T 6a813ca355f749b6b31a773aa0f2ac7a P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 3.770s I1031 23:04:37.300235 34067 raft_consensus.cc:435] T 6a813ca355f749b6b31a773aa0f2ac7a P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a8bbcbbb5a6459aa5258464755db3bb" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st02.bj" port: 18700 } } peers { permanent_uuid: "0f53a0d3ef7e44ebb0365c800752d5bd" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st01.bj" port: 18700 } } peers { permanent_uuid: "193181be29284287b25df2ee92938f0b" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st08.bj" port: 18700 } } I1031 23:04:37.300760 34067 leader_election.cc:216] T 6a813ca355f749b6b31a773aa0f2ac7a P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 0a8bbcbbb5a6459aa5258464755db3bb I1031 23:04:37.300798 34067 leader_election.cc:216] T 6a813ca355f749b6b31a773aa0f2ac7a P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 193181be29284287b25df2ee92938f0b W1031 23:04:37.301167 5816 leader_election.cc:278] T 6a813ca355f749b6b31a773aa0f2ac7a P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0a8bbcbbb5a6459aa5258464755db3bb: Illegal state: Tablet not RUNNING: NOT_STARTED W1031 23:04:37.301206 5815 leader_election.cc:278] T 6a813ca355f749b6b31a773aa0f2ac7a P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 193181be29284287b25df2ee92938f0b: Illegal state: Tablet not RUNNING: NOT_STARTED I1031 23:04:37.301228 5815 leader_election.cc:243] T 6a813ca355f749b6b31a773aa0f2ac7a P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. I1031 23:04:37.301635 34679 raft_consensus.cc:2162] T 6a813ca355f749b6b31a773aa0f2ac7a P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 16.497s I1031 23:04:37.301657 34679 raft_consensus.cc:1908] T 6a813ca355f749b6b31a773aa0f2ac7a P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority W1031 23:04:37.339992 5816 consensus_peers.cc:357] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 0a8bbcbbb5a6459aa5258464755db3bb (c3-kudu-tst-st02.bj:18700): Couldn't send request to peer 0a8bbcbbb5a6459aa5258464755db3bb for tablet 648eb376255a4dd4bb5818262efcc813. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 89 times. W1031 23:04:37.572569 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 29 times. W1031 23:04:37.587712 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 89 times. W1031 23:04:37.840014 5816 consensus_peers.cc:357] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 0a8bbcbbb5a6459aa5258464755db3bb (c3-kudu-tst-st02.bj:18700): Couldn't send request to peer 0a8bbcbbb5a6459aa5258464755db3bb for tablet 648eb376255a4dd4bb5818262efcc813. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 90 times. I1031 23:04:37.907773 34574 raft_consensus.cc:411] T bff7dab34ccd4b42842077e82106f6fb P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I1031 23:04:37.907812 34574 raft_consensus.cc:2162] T bff7dab34ccd4b42842077e82106f6fb P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 3.207s I1031 23:04:37.907865 34574 raft_consensus.cc:435] T bff7dab34ccd4b42842077e82106f6fb P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f53a0d3ef7e44ebb0365c800752d5bd" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st01.bj" port: 18700 } } peers { permanent_uuid: "23f962e4a1744381ad5fa0d2d8b10241" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st07.bj" port: 18700 } } peers { permanent_uuid: "193181be29284287b25df2ee92938f0b" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st08.bj" port: 18700 } } I1031 23:04:37.908505 34574 leader_election.cc:216] T bff7dab34ccd4b42842077e82106f6fb P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 23f962e4a1744381ad5fa0d2d8b10241 I1031 23:04:37.908540 34574 leader_election.cc:216] T bff7dab34ccd4b42842077e82106f6fb P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 193181be29284287b25df2ee92938f0b W1031 23:04:37.908819 5815 leader_election.cc:278] T bff7dab34ccd4b42842077e82106f6fb P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 193181be29284287b25df2ee92938f0b: Illegal state: Tablet not RUNNING: NOT_STARTED W1031 23:04:37.908885 5816 leader_election.cc:278] T bff7dab34ccd4b42842077e82106f6fb P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 23f962e4a1744381ad5fa0d2d8b10241: Illegal state: Tablet not RUNNING: NOT_STARTED I1031 23:04:37.908921 5816 leader_election.cc:243] T bff7dab34ccd4b42842077e82106f6fb P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. I1031 23:04:37.909271 34680 raft_consensus.cc:2162] T bff7dab34ccd4b42842077e82106f6fb P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 4.203s I1031 23:04:37.909312 34680 raft_consensus.cc:1908] T bff7dab34ccd4b42842077e82106f6fb P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority W1031 23:04:38.072624 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 30 times. W1031 23:04:38.087846 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 90 times. W1031 23:04:38.340181 5816 consensus_peers.cc:357] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 0a8bbcbbb5a6459aa5258464755db3bb (c3-kudu-tst-st02.bj:18700): Couldn't send request to peer 0a8bbcbbb5a6459aa5258464755db3bb for tablet 648eb376255a4dd4bb5818262efcc813. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 91 times. W1031 23:04:38.572849 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 31 times. W1031 23:04:38.587939 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 91 times. W1031 23:04:38.840157 5816 consensus_peers.cc:357] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 0a8bbcbbb5a6459aa5258464755db3bb (c3-kudu-tst-st02.bj:18700): Couldn't send request to peer 0a8bbcbbb5a6459aa5258464755db3bb for tablet 648eb376255a4dd4bb5818262efcc813. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 92 times. W1031 23:04:39.072789 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 32 times. W1031 23:04:39.087975 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 92 times. W1031 23:04:39.340209 5816 consensus_peers.cc:357] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 0a8bbcbbb5a6459aa5258464755db3bb (c3-kudu-tst-st02.bj:18700): Couldn't send request to peer 0a8bbcbbb5a6459aa5258464755db3bb for tablet 648eb376255a4dd4bb5818262efcc813. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 93 times. W1031 23:04:39.572917 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 33 times. W1031 23:04:39.588163 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 93 times. W1031 23:04:39.840265 5816 consensus_peers.cc:357] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 0a8bbcbbb5a6459aa5258464755db3bb (c3-kudu-tst-st02.bj:18700): Couldn't send request to peer 0a8bbcbbb5a6459aa5258464755db3bb for tablet 648eb376255a4dd4bb5818262efcc813. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 94 times. W1031 23:04:40.072950 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 34 times. W1031 23:04:40.088256 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 94 times. W1031 23:04:40.340369 5816 consensus_peers.cc:357] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 0a8bbcbbb5a6459aa5258464755db3bb (c3-kudu-tst-st02.bj:18700): Couldn't send request to peer 0a8bbcbbb5a6459aa5258464755db3bb for tablet 648eb376255a4dd4bb5818262efcc813. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 95 times. W1031 23:04:40.573045 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 35 times. W1031 23:04:40.588208 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 95 times. W1031 23:04:40.840435 5816 consensus_peers.cc:357] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 0a8bbcbbb5a6459aa5258464755db3bb (c3-kudu-tst-st02.bj:18700): Couldn't send request to peer 0a8bbcbbb5a6459aa5258464755db3bb for tablet 648eb376255a4dd4bb5818262efcc813. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 96 times. W1031 23:04:41.073071 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 36 times. W1031 23:04:41.088335 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 96 times. W1031 23:04:41.340522 5816 consensus_peers.cc:357] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 0a8bbcbbb5a6459aa5258464755db3bb (c3-kudu-tst-st02.bj:18700): Couldn't send request to peer 0a8bbcbbb5a6459aa5258464755db3bb for tablet 648eb376255a4dd4bb5818262efcc813. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 97 times. W1031 23:04:41.573246 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 37 times. W1031 23:04:41.588574 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 97 times. W1031 23:04:41.840569 5816 consensus_peers.cc:357] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 0a8bbcbbb5a6459aa5258464755db3bb (c3-kudu-tst-st02.bj:18700): Couldn't send request to peer 0a8bbcbbb5a6459aa5258464755db3bb for tablet 648eb376255a4dd4bb5818262efcc813. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 98 times. W1031 23:04:42.073269 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 38 times. W1031 23:04:42.088627 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 98 times. W1031 23:04:42.340629 5816 consensus_peers.cc:357] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 0a8bbcbbb5a6459aa5258464755db3bb (c3-kudu-tst-st02.bj:18700): Couldn't send request to peer 0a8bbcbbb5a6459aa5258464755db3bb for tablet 648eb376255a4dd4bb5818262efcc813. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 99 times. I1031 23:04:42.440150 34585 raft_consensus.cc:411] T 56155a2adc24450d89e338cc955bfa18 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I1031 23:04:42.440199 34585 raft_consensus.cc:2162] T 56155a2adc24450d89e338cc955bfa18 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 6.247s I1031 23:04:42.440253 34585 raft_consensus.cc:435] T 56155a2adc24450d89e338cc955bfa18 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625f49b9554549e6a5c3a92b76e576cf" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st10.bj" port: 18700 } } peers { permanent_uuid: "193181be29284287b25df2ee92938f0b" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st08.bj" port: 18700 } } peers { permanent_uuid: "0f53a0d3ef7e44ebb0365c800752d5bd" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st01.bj" port: 18700 } } I1031 23:04:42.440773 34585 leader_election.cc:216] T 56155a2adc24450d89e338cc955bfa18 P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 625f49b9554549e6a5c3a92b76e576cf I1031 23:04:42.440801 34585 leader_election.cc:216] T 56155a2adc24450d89e338cc955bfa18 P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 193181be29284287b25df2ee92938f0b W1031 23:04:42.441066 5814 leader_election.cc:278] T 56155a2adc24450d89e338cc955bfa18 P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 625f49b9554549e6a5c3a92b76e576cf: Illegal state: Tablet not RUNNING: NOT_STARTED W1031 23:04:42.441109 5815 leader_election.cc:278] T 56155a2adc24450d89e338cc955bfa18 P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 193181be29284287b25df2ee92938f0b: Illegal state: Tablet not RUNNING: NOT_STARTED I1031 23:04:42.441131 5815 leader_election.cc:243] T 56155a2adc24450d89e338cc955bfa18 P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. I1031 23:04:42.441890 34684 raft_consensus.cc:2162] T 56155a2adc24450d89e338cc955bfa18 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 9.200s I1031 23:04:42.441915 34684 raft_consensus.cc:1908] T 56155a2adc24450d89e338cc955bfa18 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority W1031 23:04:42.573413 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 39 times. W1031 23:04:42.588654 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 99 times. W1031 23:04:42.840694 5816 consensus_peers.cc:357] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 0a8bbcbbb5a6459aa5258464755db3bb (c3-kudu-tst-st02.bj:18700): Couldn't send request to peer 0a8bbcbbb5a6459aa5258464755db3bb for tablet 648eb376255a4dd4bb5818262efcc813. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 100 times. W1031 23:04:43.073433 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 40 times. W1031 23:04:43.088691 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 100 times. W1031 23:04:43.340788 5816 consensus_peers.cc:357] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 0a8bbcbbb5a6459aa5258464755db3bb (c3-kudu-tst-st02.bj:18700): Couldn't send request to peer 0a8bbcbbb5a6459aa5258464755db3bb for tablet 648eb376255a4dd4bb5818262efcc813. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 101 times. W1031 23:04:43.573587 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 41 times. W1031 23:04:43.588798 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 101 times. I1031 23:04:43.748692 34574 raft_consensus.cc:411] T bff7dab34ccd4b42842077e82106f6fb P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I1031 23:04:43.748742 34574 raft_consensus.cc:2162] T bff7dab34ccd4b42842077e82106f6fb P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 6.285s I1031 23:04:43.748798 34574 raft_consensus.cc:435] T bff7dab34ccd4b42842077e82106f6fb P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f53a0d3ef7e44ebb0365c800752d5bd" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st01.bj" port: 18700 } } peers { permanent_uuid: "23f962e4a1744381ad5fa0d2d8b10241" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st07.bj" port: 18700 } } peers { permanent_uuid: "193181be29284287b25df2ee92938f0b" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st08.bj" port: 18700 } } I1031 23:04:43.749500 34574 leader_election.cc:216] T bff7dab34ccd4b42842077e82106f6fb P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 23f962e4a1744381ad5fa0d2d8b10241 I1031 23:04:43.749532 34574 leader_election.cc:216] T bff7dab34ccd4b42842077e82106f6fb P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 193181be29284287b25df2ee92938f0b W1031 23:04:43.749851 5816 leader_election.cc:278] T bff7dab34ccd4b42842077e82106f6fb P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 23f962e4a1744381ad5fa0d2d8b10241: Illegal state: Tablet not RUNNING: NOT_STARTED W1031 23:04:43.749917 5815 leader_election.cc:278] T bff7dab34ccd4b42842077e82106f6fb P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 193181be29284287b25df2ee92938f0b: Illegal state: Tablet not RUNNING: NOT_STARTED I1031 23:04:43.749954 5815 leader_election.cc:243] T bff7dab34ccd4b42842077e82106f6fb P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. I1031 23:04:43.750216 34686 raft_consensus.cc:2162] T bff7dab34ccd4b42842077e82106f6fb P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 9.452s I1031 23:04:43.750236 34686 raft_consensus.cc:1908] T bff7dab34ccd4b42842077e82106f6fb P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority W1031 23:04:43.840816 5816 consensus_peers.cc:357] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 0a8bbcbbb5a6459aa5258464755db3bb (c3-kudu-tst-st02.bj:18700): Couldn't send request to peer 0a8bbcbbb5a6459aa5258464755db3bb for tablet 648eb376255a4dd4bb5818262efcc813. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 102 times. W1031 23:04:44.073571 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 42 times. W1031 23:04:44.088917 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 102 times. W1031 23:04:44.340936 5816 consensus_peers.cc:357] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 0a8bbcbbb5a6459aa5258464755db3bb (c3-kudu-tst-st02.bj:18700): Couldn't send request to peer 0a8bbcbbb5a6459aa5258464755db3bb for tablet 648eb376255a4dd4bb5818262efcc813. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 103 times. W1031 23:04:44.573705 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 43 times. W1031 23:04:44.589125 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 103 times. W1031 23:04:44.841024 5816 consensus_peers.cc:357] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 0a8bbcbbb5a6459aa5258464755db3bb (c3-kudu-tst-st02.bj:18700): Couldn't send request to peer 0a8bbcbbb5a6459aa5258464755db3bb for tablet 648eb376255a4dd4bb5818262efcc813. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 104 times. W1031 23:04:45.073714 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 44 times. W1031 23:04:45.089089 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 104 times. W1031 23:04:45.341068 5816 consensus_peers.cc:357] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 0a8bbcbbb5a6459aa5258464755db3bb (c3-kudu-tst-st02.bj:18700): Couldn't send request to peer 0a8bbcbbb5a6459aa5258464755db3bb for tablet 648eb376255a4dd4bb5818262efcc813. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 105 times. I1031 23:04:45.496393 34590 raft_consensus.cc:411] T 117cd74bcdab4942901acb0005cdd702 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I1031 23:04:45.496438 34590 raft_consensus.cc:2162] T 117cd74bcdab4942901acb0005cdd702 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.540s I1031 23:04:45.496508 34590 raft_consensus.cc:435] T 117cd74bcdab4942901acb0005cdd702 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625f49b9554549e6a5c3a92b76e576cf" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st10.bj" port: 18700 } } peers { permanent_uuid: "0a8bbcbbb5a6459aa5258464755db3bb" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st02.bj" port: 18700 } } peers { permanent_uuid: "0f53a0d3ef7e44ebb0365c800752d5bd" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st01.bj" port: 18700 } } I1031 23:04:45.497242 34590 leader_election.cc:216] T 117cd74bcdab4942901acb0005cdd702 P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 625f49b9554549e6a5c3a92b76e576cf I1031 23:04:45.497284 34590 leader_election.cc:216] T 117cd74bcdab4942901acb0005cdd702 P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 0a8bbcbbb5a6459aa5258464755db3bb W1031 23:04:45.497524 5814 leader_election.cc:278] T 117cd74bcdab4942901acb0005cdd702 P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 625f49b9554549e6a5c3a92b76e576cf: Illegal state: Tablet not RUNNING: NOT_STARTED W1031 23:04:45.497568 5816 leader_election.cc:278] T 117cd74bcdab4942901acb0005cdd702 P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0a8bbcbbb5a6459aa5258464755db3bb: Illegal state: Tablet not RUNNING: NOT_STARTED I1031 23:04:45.497593 5816 leader_election.cc:243] T 117cd74bcdab4942901acb0005cdd702 P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. I1031 23:04:45.497874 34705 raft_consensus.cc:2162] T 117cd74bcdab4942901acb0005cdd702 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 7.236s I1031 23:04:45.497890 34705 raft_consensus.cc:1908] T 117cd74bcdab4942901acb0005cdd702 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority W1031 23:04:45.573750 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 45 times. W1031 23:04:45.589148 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 105 times. W1031 23:04:45.841109 5816 consensus_peers.cc:357] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 0a8bbcbbb5a6459aa5258464755db3bb (c3-kudu-tst-st02.bj:18700): Couldn't send request to peer 0a8bbcbbb5a6459aa5258464755db3bb for tablet 648eb376255a4dd4bb5818262efcc813. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 106 times. W1031 23:04:46.073873 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 46 times. W1031 23:04:46.089248 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 106 times. W1031 23:04:46.341269 5816 consensus_peers.cc:357] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 0a8bbcbbb5a6459aa5258464755db3bb (c3-kudu-tst-st02.bj:18700): Couldn't send request to peer 0a8bbcbbb5a6459aa5258464755db3bb for tablet 648eb376255a4dd4bb5818262efcc813. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 107 times. W1031 23:04:46.573930 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 47 times. W1031 23:04:46.589229 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 107 times. W1031 23:04:46.841246 5816 consensus_peers.cc:357] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 0a8bbcbbb5a6459aa5258464755db3bb (c3-kudu-tst-st02.bj:18700): Couldn't send request to peer 0a8bbcbbb5a6459aa5258464755db3bb for tablet 648eb376255a4dd4bb5818262efcc813. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 108 times. W1031 23:04:47.073984 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 48 times. W1031 23:04:47.089617 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 108 times. W1031 23:04:47.341513 5816 consensus_peers.cc:357] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 0a8bbcbbb5a6459aa5258464755db3bb (c3-kudu-tst-st02.bj:18700): Couldn't send request to peer 0a8bbcbbb5a6459aa5258464755db3bb for tablet 648eb376255a4dd4bb5818262efcc813. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 109 times. W1031 23:04:47.574133 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 49 times. W1031 23:04:47.589500 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 109 times. W1031 23:04:47.841439 5816 consensus_peers.cc:357] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 0a8bbcbbb5a6459aa5258464755db3bb (c3-kudu-tst-st02.bj:18700): Couldn't send request to peer 0a8bbcbbb5a6459aa5258464755db3bb for tablet 648eb376255a4dd4bb5818262efcc813. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 110 times. W1031 23:04:48.074160 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 50 times. W1031 23:04:48.089522 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 110 times. W1031 23:04:48.341488 5816 consensus_peers.cc:357] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 0a8bbcbbb5a6459aa5258464755db3bb (c3-kudu-tst-st02.bj:18700): Couldn't send request to peer 0a8bbcbbb5a6459aa5258464755db3bb for tablet 648eb376255a4dd4bb5818262efcc813. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 111 times. W1031 23:04:48.574327 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 51 times. W1031 23:04:48.589604 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 111 times. W1031 23:04:48.841570 5816 consensus_peers.cc:357] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 0a8bbcbbb5a6459aa5258464755db3bb (c3-kudu-tst-st02.bj:18700): Couldn't send request to peer 0a8bbcbbb5a6459aa5258464755db3bb for tablet 648eb376255a4dd4bb5818262efcc813. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 112 times. W1031 23:04:49.074339 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 52 times. W1031 23:04:49.089645 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 112 times. W1031 23:04:49.341634 5816 consensus_peers.cc:357] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 0a8bbcbbb5a6459aa5258464755db3bb (c3-kudu-tst-st02.bj:18700): Couldn't send request to peer 0a8bbcbbb5a6459aa5258464755db3bb for tablet 648eb376255a4dd4bb5818262efcc813. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 113 times. W1031 23:04:49.574357 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 53 times. W1031 23:04:49.589757 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 113 times. W1031 23:04:49.841681 5816 consensus_peers.cc:357] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 0a8bbcbbb5a6459aa5258464755db3bb (c3-kudu-tst-st02.bj:18700): Couldn't send request to peer 0a8bbcbbb5a6459aa5258464755db3bb for tablet 648eb376255a4dd4bb5818262efcc813. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 114 times. W1031 23:04:50.074429 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 54 times. W1031 23:04:50.089800 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 114 times. W1031 23:04:50.341759 5816 consensus_peers.cc:357] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 0a8bbcbbb5a6459aa5258464755db3bb (c3-kudu-tst-st02.bj:18700): Couldn't send request to peer 0a8bbcbbb5a6459aa5258464755db3bb for tablet 648eb376255a4dd4bb5818262efcc813. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 115 times. W1031 23:04:50.574590 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 55 times. W1031 23:04:50.589910 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 115 times. W1031 23:04:50.841838 5816 consensus_peers.cc:357] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 0a8bbcbbb5a6459aa5258464755db3bb (c3-kudu-tst-st02.bj:18700): Couldn't send request to peer 0a8bbcbbb5a6459aa5258464755db3bb for tablet 648eb376255a4dd4bb5818262efcc813. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 116 times. W1031 23:04:51.074570 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 56 times. W1031 23:04:51.090052 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 116 times. W1031 23:04:51.342093 5816 consensus_peers.cc:357] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 0a8bbcbbb5a6459aa5258464755db3bb (c3-kudu-tst-st02.bj:18700): Couldn't send request to peer 0a8bbcbbb5a6459aa5258464755db3bb for tablet 648eb376255a4dd4bb5818262efcc813. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 117 times. W1031 23:04:51.574580 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 57 times. W1031 23:04:51.589975 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 117 times. W1031 23:04:51.841939 5816 consensus_peers.cc:357] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 0a8bbcbbb5a6459aa5258464755db3bb (c3-kudu-tst-st02.bj:18700): Couldn't send request to peer 0a8bbcbbb5a6459aa5258464755db3bb for tablet 648eb376255a4dd4bb5818262efcc813. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 118 times. W1031 23:04:52.074764 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 58 times. W1031 23:04:52.090212 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 118 times. W1031 23:04:52.342052 5816 consensus_peers.cc:357] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 0a8bbcbbb5a6459aa5258464755db3bb (c3-kudu-tst-st02.bj:18700): Couldn't send request to peer 0a8bbcbbb5a6459aa5258464755db3bb for tablet 648eb376255a4dd4bb5818262efcc813. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 119 times. I1031 23:04:52.500762 5866 tablet_service.cc:691] Processing DeleteTablet for tablet bff7dab34ccd4b42842077e82106f6fb with delete_type TABLET_DATA_DELETED (Replaced by 005a48fb466c4a52896842dd00124755 at 2017-10-31 23:04:52 CST) from {username='work'} at 10.108.187.10:53997 I1031 23:04:52.500999 5857 tablet_service.cc:691] Processing DeleteTablet for tablet 56155a2adc24450d89e338cc955bfa18 with delete_type TABLET_DATA_DELETED (Replaced by 80ba7f33ec864c5f96c61932edb5cdc4 at 2017-10-31 23:04:52 CST) from {username='work'} at 10.108.187.10:53997 I1031 23:04:52.501253 5857 tablet_service.cc:691] Processing DeleteTablet for tablet 117cd74bcdab4942901acb0005cdd702 with delete_type TABLET_DATA_DELETED (Replaced by acb401e40e7049fbbfdbeb40474353d9 at 2017-10-31 23:04:52 CST) from {username='work'} at 10.108.187.10:53997 I1031 23:04:52.517606 5857 tablet_service.cc:691] Processing DeleteTablet for tablet 117cd74bcdab4942901acb0005cdd702 with delete_type TABLET_DATA_DELETED (Replaced by acb401e40e7049fbbfdbeb40474353d9 at 2017-10-31 23:04:52 CST) from {username='work'} at 10.108.187.10:53997 I1031 23:04:52.538297 5857 tablet_service.cc:691] Processing DeleteTablet for tablet 6a813ca355f749b6b31a773aa0f2ac7a with delete_type TABLET_DATA_DELETED (Replaced by 582c1281bf5147b2bd40c9a950a1152f at 2017-10-31 23:04:22 CST) from {username='work'} at 10.108.187.10:53997 I1031 23:04:52.538313 5866 tablet_service.cc:691] Processing DeleteTablet for tablet 56155a2adc24450d89e338cc955bfa18 with delete_type TABLET_DATA_DELETED (Replaced by 80ba7f33ec864c5f96c61932edb5cdc4 at 2017-10-31 23:04:52 CST) from {username='work'} at 10.108.187.10:53997 I1031 23:04:52.553421 5866 tablet_service.cc:691] Processing DeleteTablet for tablet 117cd74bcdab4942901acb0005cdd702 with delete_type TABLET_DATA_DELETED (Replaced by acb401e40e7049fbbfdbeb40474353d9 at 2017-10-31 23:04:52 CST) from {username='work'} at 10.108.187.10:53997 I1031 23:04:52.565732 5866 tablet_service.cc:691] Processing DeleteTablet for tablet bff7dab34ccd4b42842077e82106f6fb with delete_type TABLET_DATA_DELETED (Replaced by 005a48fb466c4a52896842dd00124755 at 2017-10-31 23:04:52 CST) from {username='work'} at 10.108.187.10:53997 W1031 23:04:52.574772 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 59 times. I1031 23:04:52.581990 5866 tablet_service.cc:691] Processing DeleteTablet for tablet 6a813ca355f749b6b31a773aa0f2ac7a with delete_type TABLET_DATA_DELETED (Replaced by 582c1281bf5147b2bd40c9a950a1152f at 2017-10-31 23:04:22 CST) from {username='work'} at 10.108.187.10:53997 I1031 23:04:52.582047 5857 tablet_service.cc:691] Processing DeleteTablet for tablet 56155a2adc24450d89e338cc955bfa18 with delete_type TABLET_DATA_DELETED (Replaced by 80ba7f33ec864c5f96c61932edb5cdc4 at 2017-10-31 23:04:52 CST) from {username='work'} at 10.108.187.10:53997 W1031 23:04:52.590147 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 119 times. I1031 23:04:52.628715 5857 tablet_service.cc:691] Processing DeleteTablet for tablet 117cd74bcdab4942901acb0005cdd702 with delete_type TABLET_DATA_DELETED (Replaced by acb401e40e7049fbbfdbeb40474353d9 at 2017-10-31 23:04:52 CST) from {username='work'} at 10.108.187.10:53997 I1031 23:04:52.643260 5857 tablet_service.cc:691] Processing DeleteTablet for tablet bff7dab34ccd4b42842077e82106f6fb with delete_type TABLET_DATA_DELETED (Replaced by 005a48fb466c4a52896842dd00124755 at 2017-10-31 23:04:52 CST) from {username='work'} at 10.108.187.10:53997 I1031 23:04:52.656635 5857 tablet_service.cc:691] Processing DeleteTablet for tablet 6a813ca355f749b6b31a773aa0f2ac7a with delete_type TABLET_DATA_DELETED (Replaced by 582c1281bf5147b2bd40c9a950a1152f at 2017-10-31 23:04:22 CST) from {username='work'} at 10.108.187.10:53997 I1031 23:04:52.680060 5857 tablet_service.cc:691] Processing DeleteTablet for tablet 56155a2adc24450d89e338cc955bfa18 with delete_type TABLET_DATA_DELETED (Replaced by 80ba7f33ec864c5f96c61932edb5cdc4 at 2017-10-31 23:04:52 CST) from {username='work'} at 10.108.187.10:53997 I1031 23:04:52.745702 5857 tablet_service.cc:691] Processing DeleteTablet for tablet bff7dab34ccd4b42842077e82106f6fb with delete_type TABLET_DATA_DELETED (Replaced by 005a48fb466c4a52896842dd00124755 at 2017-10-31 23:04:52 CST) from {username='work'} at 10.108.187.10:53997 I1031 23:04:52.795684 5857 tablet_service.cc:691] Processing DeleteTablet for tablet 117cd74bcdab4942901acb0005cdd702 with delete_type TABLET_DATA_DELETED (Replaced by acb401e40e7049fbbfdbeb40474353d9 at 2017-10-31 23:04:52 CST) from {username='work'} at 10.108.187.10:53997 I1031 23:04:52.828263 5857 tablet_service.cc:691] Processing DeleteTablet for tablet 6a813ca355f749b6b31a773aa0f2ac7a with delete_type TABLET_DATA_DELETED (Replaced by 582c1281bf5147b2bd40c9a950a1152f at 2017-10-31 23:04:22 CST) from {username='work'} at 10.108.187.10:53997 I1031 23:04:52.839578 5857 tablet_service.cc:691] Processing DeleteTablet for tablet 56155a2adc24450d89e338cc955bfa18 with delete_type TABLET_DATA_DELETED (Replaced by 80ba7f33ec864c5f96c61932edb5cdc4 at 2017-10-31 23:04:52 CST) from {username='work'} at 10.108.187.10:53997 W1031 23:04:52.842090 5816 consensus_peers.cc:357] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 0a8bbcbbb5a6459aa5258464755db3bb (c3-kudu-tst-st02.bj:18700): Couldn't send request to peer 0a8bbcbbb5a6459aa5258464755db3bb for tablet 648eb376255a4dd4bb5818262efcc813. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 120 times. I1031 23:04:52.878057 5857 tablet_service.cc:691] Processing DeleteTablet for tablet bff7dab34ccd4b42842077e82106f6fb with delete_type TABLET_DATA_DELETED (Replaced by 005a48fb466c4a52896842dd00124755 at 2017-10-31 23:04:52 CST) from {username='work'} at 10.108.187.10:53997 W1031 23:04:53.074895 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 60 times. I1031 23:04:53.089522 5857 tablet_service.cc:691] Processing DeleteTablet for tablet 117cd74bcdab4942901acb0005cdd702 with delete_type TABLET_DATA_DELETED (Replaced by acb401e40e7049fbbfdbeb40474353d9 at 2017-10-31 23:04:52 CST) from {username='work'} at 10.108.187.10:53997 W1031 23:04:53.090282 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 120 times. I1031 23:04:53.120851 5857 tablet_service.cc:691] Processing DeleteTablet for tablet 56155a2adc24450d89e338cc955bfa18 with delete_type TABLET_DATA_DELETED (Replaced by 80ba7f33ec864c5f96c61932edb5cdc4 at 2017-10-31 23:04:52 CST) from {username='work'} at 10.108.187.10:53997 I1031 23:04:53.125296 5857 tablet_service.cc:691] Processing DeleteTablet for tablet 6a813ca355f749b6b31a773aa0f2ac7a with delete_type TABLET_DATA_DELETED (Replaced by 582c1281bf5147b2bd40c9a950a1152f at 2017-10-31 23:04:22 CST) from {username='work'} at 10.108.187.10:53997 I1031 23:04:53.149552 5857 tablet_service.cc:691] Processing DeleteTablet for tablet bff7dab34ccd4b42842077e82106f6fb with delete_type TABLET_DATA_DELETED (Replaced by 005a48fb466c4a52896842dd00124755 at 2017-10-31 23:04:52 CST) from {username='work'} at 10.108.187.10:53997 W1031 23:04:53.342234 5816 consensus_peers.cc:357] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 0a8bbcbbb5a6459aa5258464755db3bb (c3-kudu-tst-st02.bj:18700): Couldn't send request to peer 0a8bbcbbb5a6459aa5258464755db3bb for tablet 648eb376255a4dd4bb5818262efcc813. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 121 times. I1031 23:04:53.378113 34585 raft_consensus.cc:411] T 56155a2adc24450d89e338cc955bfa18 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I1031 23:04:53.378168 34585 raft_consensus.cc:2162] T 56155a2adc24450d89e338cc955bfa18 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 2.859s I1031 23:04:53.378226 34585 raft_consensus.cc:435] T 56155a2adc24450d89e338cc955bfa18 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625f49b9554549e6a5c3a92b76e576cf" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st10.bj" port: 18700 } } peers { permanent_uuid: "193181be29284287b25df2ee92938f0b" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st08.bj" port: 18700 } } peers { permanent_uuid: "0f53a0d3ef7e44ebb0365c800752d5bd" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st01.bj" port: 18700 } } I1031 23:04:53.378964 34585 leader_election.cc:216] T 56155a2adc24450d89e338cc955bfa18 P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 625f49b9554549e6a5c3a92b76e576cf I1031 23:04:53.378993 34585 leader_election.cc:216] T 56155a2adc24450d89e338cc955bfa18 P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 193181be29284287b25df2ee92938f0b W1031 23:04:53.379294 5814 leader_election.cc:278] T 56155a2adc24450d89e338cc955bfa18 P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 625f49b9554549e6a5c3a92b76e576cf: Illegal state: Tablet not RUNNING: NOT_STARTED W1031 23:04:53.379429 5815 leader_election.cc:278] T 56155a2adc24450d89e338cc955bfa18 P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 193181be29284287b25df2ee92938f0b: Illegal state: Tablet not RUNNING: NOT_STARTED I1031 23:04:53.379454 5815 leader_election.cc:243] T 56155a2adc24450d89e338cc955bfa18 P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. I1031 23:04:53.379709 34715 raft_consensus.cc:2162] T 56155a2adc24450d89e338cc955bfa18 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 7.570s I1031 23:04:53.379729 34715 raft_consensus.cc:1908] T 56155a2adc24450d89e338cc955bfa18 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority W1031 23:04:53.575058 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 61 times. W1031 23:04:53.590329 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 121 times. I1031 23:04:53.607214 5857 tablet_service.cc:691] Processing DeleteTablet for tablet 117cd74bcdab4942901acb0005cdd702 with delete_type TABLET_DATA_DELETED (Replaced by acb401e40e7049fbbfdbeb40474353d9 at 2017-10-31 23:04:52 CST) from {username='work'} at 10.108.187.10:53997 I1031 23:04:53.662652 5857 tablet_service.cc:691] Processing DeleteTablet for tablet 6a813ca355f749b6b31a773aa0f2ac7a with delete_type TABLET_DATA_DELETED (Replaced by 582c1281bf5147b2bd40c9a950a1152f at 2017-10-31 23:04:22 CST) from {username='work'} at 10.108.187.10:53997 I1031 23:04:53.675498 5857 tablet_service.cc:691] Processing DeleteTablet for tablet 56155a2adc24450d89e338cc955bfa18 with delete_type TABLET_DATA_DELETED (Replaced by 80ba7f33ec864c5f96c61932edb5cdc4 at 2017-10-31 23:04:52 CST) from {username='work'} at 10.108.187.10:53997 I1031 23:04:53.683866 5857 tablet_service.cc:691] Processing DeleteTablet for tablet bff7dab34ccd4b42842077e82106f6fb with delete_type TABLET_DATA_DELETED (Replaced by 005a48fb466c4a52896842dd00124755 at 2017-10-31 23:04:52 CST) from {username='work'} at 10.108.187.10:53997 W1031 23:04:53.842236 5816 consensus_peers.cc:357] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 0a8bbcbbb5a6459aa5258464755db3bb (c3-kudu-tst-st02.bj:18700): Couldn't send request to peer 0a8bbcbbb5a6459aa5258464755db3bb for tablet 648eb376255a4dd4bb5818262efcc813. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 122 times. W1031 23:04:54.075036 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 62 times. W1031 23:04:54.090435 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 122 times. I1031 23:04:54.338776 34590 raft_consensus.cc:411] T 117cd74bcdab4942901acb0005cdd702 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I1031 23:04:54.338820 34590 raft_consensus.cc:2162] T 117cd74bcdab4942901acb0005cdd702 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 10.815s I1031 23:04:54.338882 34590 raft_consensus.cc:435] T 117cd74bcdab4942901acb0005cdd702 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625f49b9554549e6a5c3a92b76e576cf" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st10.bj" port: 18700 } } peers { permanent_uuid: "0a8bbcbbb5a6459aa5258464755db3bb" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st02.bj" port: 18700 } } peers { permanent_uuid: "0f53a0d3ef7e44ebb0365c800752d5bd" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st01.bj" port: 18700 } } I1031 23:04:54.339679 34590 leader_election.cc:216] T 117cd74bcdab4942901acb0005cdd702 P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 625f49b9554549e6a5c3a92b76e576cf I1031 23:04:54.339711 34590 leader_election.cc:216] T 117cd74bcdab4942901acb0005cdd702 P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 0a8bbcbbb5a6459aa5258464755db3bb W1031 23:04:54.339937 5816 leader_election.cc:278] T 117cd74bcdab4942901acb0005cdd702 P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0a8bbcbbb5a6459aa5258464755db3bb: Illegal state: Tablet not RUNNING: NOT_STARTED W1031 23:04:54.339970 5814 leader_election.cc:278] T 117cd74bcdab4942901acb0005cdd702 P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 625f49b9554549e6a5c3a92b76e576cf: Illegal state: Tablet not RUNNING: NOT_STARTED I1031 23:04:54.339993 5814 leader_election.cc:243] T 117cd74bcdab4942901acb0005cdd702 P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. I1031 23:04:54.340778 34720 raft_consensus.cc:2162] T 117cd74bcdab4942901acb0005cdd702 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 12.247s I1031 23:04:54.340801 34720 raft_consensus.cc:1908] T 117cd74bcdab4942901acb0005cdd702 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority W1031 23:04:54.342366 5816 consensus_peers.cc:357] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 0a8bbcbbb5a6459aa5258464755db3bb (c3-kudu-tst-st02.bj:18700): Couldn't send request to peer 0a8bbcbbb5a6459aa5258464755db3bb for tablet 648eb376255a4dd4bb5818262efcc813. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 123 times. W1031 23:04:54.575255 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 63 times. W1031 23:04:54.590474 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 123 times. I1031 23:04:54.664094 5857 tablet_service.cc:691] Processing DeleteTablet for tablet 117cd74bcdab4942901acb0005cdd702 with delete_type TABLET_DATA_DELETED (Replaced by acb401e40e7049fbbfdbeb40474353d9 at 2017-10-31 23:04:52 CST) from {username='work'} at 10.108.187.10:53997 I1031 23:04:54.708130 5857 tablet_service.cc:691] Processing DeleteTablet for tablet 56155a2adc24450d89e338cc955bfa18 with delete_type TABLET_DATA_DELETED (Replaced by 80ba7f33ec864c5f96c61932edb5cdc4 at 2017-10-31 23:04:52 CST) from {username='work'} at 10.108.187.10:53997 I1031 23:04:54.713534 5857 tablet_service.cc:691] Processing DeleteTablet for tablet 6a813ca355f749b6b31a773aa0f2ac7a with delete_type TABLET_DATA_DELETED (Replaced by 582c1281bf5147b2bd40c9a950a1152f at 2017-10-31 23:04:22 CST) from {username='work'} at 10.108.187.10:53997 I1031 23:04:54.737401 5857 tablet_service.cc:691] Processing DeleteTablet for tablet bff7dab34ccd4b42842077e82106f6fb with delete_type TABLET_DATA_DELETED (Replaced by 005a48fb466c4a52896842dd00124755 at 2017-10-31 23:04:52 CST) from {username='work'} at 10.108.187.10:53997 W1031 23:04:54.842366 5816 consensus_peers.cc:357] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 0a8bbcbbb5a6459aa5258464755db3bb (c3-kudu-tst-st02.bj:18700): Couldn't send request to peer 0a8bbcbbb5a6459aa5258464755db3bb for tablet 648eb376255a4dd4bb5818262efcc813. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 124 times. W1031 23:04:55.075150 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 64 times. W1031 23:04:55.090570 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 124 times. I1031 23:04:55.091480 34574 raft_consensus.cc:411] T bff7dab34ccd4b42842077e82106f6fb P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I1031 23:04:55.091518 34574 raft_consensus.cc:2162] T bff7dab34ccd4b42842077e82106f6fb P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 15.945s I1031 23:04:55.091569 34574 raft_consensus.cc:435] T bff7dab34ccd4b42842077e82106f6fb P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f53a0d3ef7e44ebb0365c800752d5bd" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st01.bj" port: 18700 } } peers { permanent_uuid: "23f962e4a1744381ad5fa0d2d8b10241" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st07.bj" port: 18700 } } peers { permanent_uuid: "193181be29284287b25df2ee92938f0b" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st08.bj" port: 18700 } } I1031 23:04:55.092141 34574 leader_election.cc:216] T bff7dab34ccd4b42842077e82106f6fb P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 23f962e4a1744381ad5fa0d2d8b10241 I1031 23:04:55.092187 34574 leader_election.cc:216] T bff7dab34ccd4b42842077e82106f6fb P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 193181be29284287b25df2ee92938f0b W1031 23:04:55.092453 5815 leader_election.cc:278] T bff7dab34ccd4b42842077e82106f6fb P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 193181be29284287b25df2ee92938f0b: Illegal state: Tablet not RUNNING: NOT_STARTED W1031 23:04:55.092485 5816 leader_election.cc:278] T bff7dab34ccd4b42842077e82106f6fb P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 23f962e4a1744381ad5fa0d2d8b10241: Illegal state: Tablet not RUNNING: NOT_STARTED I1031 23:04:55.092520 5816 leader_election.cc:243] T bff7dab34ccd4b42842077e82106f6fb P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. I1031 23:04:55.092726 34737 raft_consensus.cc:2162] T bff7dab34ccd4b42842077e82106f6fb P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 6.796s I1031 23:04:55.092748 34737 raft_consensus.cc:1908] T bff7dab34ccd4b42842077e82106f6fb P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority W1031 23:04:55.342483 5816 consensus_peers.cc:357] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 0a8bbcbbb5a6459aa5258464755db3bb (c3-kudu-tst-st02.bj:18700): Couldn't send request to peer 0a8bbcbbb5a6459aa5258464755db3bb for tablet 648eb376255a4dd4bb5818262efcc813. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 125 times. I1031 23:04:55.352864 34067 raft_consensus.cc:411] T 6a813ca355f749b6b31a773aa0f2ac7a P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I1031 23:04:55.352902 34067 raft_consensus.cc:2162] T 6a813ca355f749b6b31a773aa0f2ac7a P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 19.168s I1031 23:04:55.352958 34067 raft_consensus.cc:435] T 6a813ca355f749b6b31a773aa0f2ac7a P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a8bbcbbb5a6459aa5258464755db3bb" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st02.bj" port: 18700 } } peers { permanent_uuid: "0f53a0d3ef7e44ebb0365c800752d5bd" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st01.bj" port: 18700 } } peers { permanent_uuid: "193181be29284287b25df2ee92938f0b" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st08.bj" port: 18700 } } I1031 23:04:55.353478 34067 leader_election.cc:216] T 6a813ca355f749b6b31a773aa0f2ac7a P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 0a8bbcbbb5a6459aa5258464755db3bb I1031 23:04:55.353512 34067 leader_election.cc:216] T 6a813ca355f749b6b31a773aa0f2ac7a P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 193181be29284287b25df2ee92938f0b W1031 23:04:55.353762 5816 leader_election.cc:278] T 6a813ca355f749b6b31a773aa0f2ac7a P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0a8bbcbbb5a6459aa5258464755db3bb: Illegal state: Tablet not RUNNING: NOT_STARTED W1031 23:04:55.353796 5815 leader_election.cc:278] T 6a813ca355f749b6b31a773aa0f2ac7a P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 193181be29284287b25df2ee92938f0b: Illegal state: Tablet not RUNNING: NOT_STARTED I1031 23:04:55.353818 5815 leader_election.cc:243] T 6a813ca355f749b6b31a773aa0f2ac7a P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. I1031 23:04:55.354068 34738 raft_consensus.cc:2162] T 6a813ca355f749b6b31a773aa0f2ac7a P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 4.726s I1031 23:04:55.354082 34738 raft_consensus.cc:1908] T 6a813ca355f749b6b31a773aa0f2ac7a P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority W1031 23:04:55.575381 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 65 times. W1031 23:04:55.590618 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 125 times. W1031 23:04:55.842541 5816 consensus_peers.cc:357] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 0a8bbcbbb5a6459aa5258464755db3bb (c3-kudu-tst-st02.bj:18700): Couldn't send request to peer 0a8bbcbbb5a6459aa5258464755db3bb for tablet 648eb376255a4dd4bb5818262efcc813. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 126 times. W1031 23:04:56.075353 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 66 times. W1031 23:04:56.090683 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 126 times. W1031 23:04:56.342609 5816 consensus_peers.cc:357] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 0a8bbcbbb5a6459aa5258464755db3bb (c3-kudu-tst-st02.bj:18700): Couldn't send request to peer 0a8bbcbbb5a6459aa5258464755db3bb for tablet 648eb376255a4dd4bb5818262efcc813. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 127 times. W1031 23:04:56.575486 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 67 times. W1031 23:04:56.590759 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 127 times. I1031 23:04:56.716821 5857 tablet_service.cc:691] Processing DeleteTablet for tablet 117cd74bcdab4942901acb0005cdd702 with delete_type TABLET_DATA_DELETED (Replaced by acb401e40e7049fbbfdbeb40474353d9 at 2017-10-31 23:04:52 CST) from {username='work'} at 10.108.187.10:53997 I1031 23:04:56.805676 5857 tablet_service.cc:691] Processing DeleteTablet for tablet 56155a2adc24450d89e338cc955bfa18 with delete_type TABLET_DATA_DELETED (Replaced by 80ba7f33ec864c5f96c61932edb5cdc4 at 2017-10-31 23:04:52 CST) from {username='work'} at 10.108.187.10:53997 I1031 23:04:56.805963 5857 tablet_service.cc:691] Processing DeleteTablet for tablet 6a813ca355f749b6b31a773aa0f2ac7a with delete_type TABLET_DATA_DELETED (Replaced by 582c1281bf5147b2bd40c9a950a1152f at 2017-10-31 23:04:22 CST) from {username='work'} at 10.108.187.10:53997 I1031 23:04:56.809432 5857 tablet_service.cc:691] Processing DeleteTablet for tablet bff7dab34ccd4b42842077e82106f6fb with delete_type TABLET_DATA_DELETED (Replaced by 005a48fb466c4a52896842dd00124755 at 2017-10-31 23:04:52 CST) from {username='work'} at 10.108.187.10:53997 W1031 23:04:56.842669 5816 consensus_peers.cc:357] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 0a8bbcbbb5a6459aa5258464755db3bb (c3-kudu-tst-st02.bj:18700): Couldn't send request to peer 0a8bbcbbb5a6459aa5258464755db3bb for tablet 648eb376255a4dd4bb5818262efcc813. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 128 times. W1031 23:04:57.075474 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 68 times. W1031 23:04:57.090821 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 128 times. W1031 23:04:57.342751 5816 consensus_peers.cc:357] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 0a8bbcbbb5a6459aa5258464755db3bb (c3-kudu-tst-st02.bj:18700): Couldn't send request to peer 0a8bbcbbb5a6459aa5258464755db3bb for tablet 648eb376255a4dd4bb5818262efcc813. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 129 times. W1031 23:04:57.575606 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 69 times. W1031 23:04:57.590867 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 129 times. W1031 23:04:57.842823 5816 consensus_peers.cc:357] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 0a8bbcbbb5a6459aa5258464755db3bb (c3-kudu-tst-st02.bj:18700): Couldn't send request to peer 0a8bbcbbb5a6459aa5258464755db3bb for tablet 648eb376255a4dd4bb5818262efcc813. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 130 times. W1031 23:04:58.075654 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 70 times. W1031 23:04:58.090937 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 130 times. W1031 23:04:58.342859 5816 consensus_peers.cc:357] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 0a8bbcbbb5a6459aa5258464755db3bb (c3-kudu-tst-st02.bj:18700): Couldn't send request to peer 0a8bbcbbb5a6459aa5258464755db3bb for tablet 648eb376255a4dd4bb5818262efcc813. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 131 times. W1031 23:04:58.575744 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 71 times. W1031 23:04:58.591008 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 131 times. W1031 23:04:58.842891 5816 consensus_peers.cc:357] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 0a8bbcbbb5a6459aa5258464755db3bb (c3-kudu-tst-st02.bj:18700): Couldn't send request to peer 0a8bbcbbb5a6459aa5258464755db3bb for tablet 648eb376255a4dd4bb5818262efcc813. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 132 times. W1031 23:04:59.075845 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 72 times. W1031 23:04:59.091123 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 132 times. W1031 23:04:59.343153 5816 consensus_peers.cc:357] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 0a8bbcbbb5a6459aa5258464755db3bb (c3-kudu-tst-st02.bj:18700): Couldn't send request to peer 0a8bbcbbb5a6459aa5258464755db3bb for tablet 648eb376255a4dd4bb5818262efcc813. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 133 times. W1031 23:04:59.575997 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 73 times. W1031 23:04:59.591140 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 133 times. W1031 23:04:59.843087 5816 consensus_peers.cc:357] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 0a8bbcbbb5a6459aa5258464755db3bb (c3-kudu-tst-st02.bj:18700): Couldn't send request to peer 0a8bbcbbb5a6459aa5258464755db3bb for tablet 648eb376255a4dd4bb5818262efcc813. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 134 times. W1031 23:05:00.075868 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 74 times. W1031 23:05:00.091204 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 134 times. W1031 23:05:00.343153 5816 consensus_peers.cc:357] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 0a8bbcbbb5a6459aa5258464755db3bb (c3-kudu-tst-st02.bj:18700): Couldn't send request to peer 0a8bbcbbb5a6459aa5258464755db3bb for tablet 648eb376255a4dd4bb5818262efcc813. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 135 times. W1031 23:05:00.576036 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 75 times. W1031 23:05:00.591312 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 135 times. I1031 23:05:00.834950 5857 tablet_service.cc:691] Processing DeleteTablet for tablet 117cd74bcdab4942901acb0005cdd702 with delete_type TABLET_DATA_DELETED (Replaced by acb401e40e7049fbbfdbeb40474353d9 at 2017-10-31 23:04:52 CST) from {username='work'} at 10.108.187.10:53997 W1031 23:05:00.843251 5816 consensus_peers.cc:357] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 0a8bbcbbb5a6459aa5258464755db3bb (c3-kudu-tst-st02.bj:18700): Couldn't send request to peer 0a8bbcbbb5a6459aa5258464755db3bb for tablet 648eb376255a4dd4bb5818262efcc813. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 136 times. I1031 23:05:00.931903 5857 tablet_service.cc:691] Processing DeleteTablet for tablet 6a813ca355f749b6b31a773aa0f2ac7a with delete_type TABLET_DATA_DELETED (Replaced by 582c1281bf5147b2bd40c9a950a1152f at 2017-10-31 23:04:22 CST) from {username='work'} at 10.108.187.10:53997 I1031 23:05:00.940299 5857 tablet_service.cc:691] Processing DeleteTablet for tablet bff7dab34ccd4b42842077e82106f6fb with delete_type TABLET_DATA_DELETED (Replaced by 005a48fb466c4a52896842dd00124755 at 2017-10-31 23:04:52 CST) from {username='work'} at 10.108.187.10:53997 I1031 23:05:00.947742 5857 tablet_service.cc:691] Processing DeleteTablet for tablet 56155a2adc24450d89e338cc955bfa18 with delete_type TABLET_DATA_DELETED (Replaced by 80ba7f33ec864c5f96c61932edb5cdc4 at 2017-10-31 23:04:52 CST) from {username='work'} at 10.108.187.10:53997 W1031 23:05:01.076139 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 76 times. W1031 23:05:01.091392 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 136 times. W1031 23:05:01.343408 5816 consensus_peers.cc:357] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 0a8bbcbbb5a6459aa5258464755db3bb (c3-kudu-tst-st02.bj:18700): Couldn't send request to peer 0a8bbcbbb5a6459aa5258464755db3bb for tablet 648eb376255a4dd4bb5818262efcc813. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 137 times. W1031 23:05:01.576153 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 77 times. W1031 23:05:01.591524 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 137 times. I1031 23:05:01.800616 34067 raft_consensus.cc:411] T 6a813ca355f749b6b31a773aa0f2ac7a P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I1031 23:05:01.800667 34067 raft_consensus.cc:2162] T 6a813ca355f749b6b31a773aa0f2ac7a P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 11.225s I1031 23:05:01.800719 34067 raft_consensus.cc:435] T 6a813ca355f749b6b31a773aa0f2ac7a P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a8bbcbbb5a6459aa5258464755db3bb" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st02.bj" port: 18700 } } peers { permanent_uuid: "0f53a0d3ef7e44ebb0365c800752d5bd" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st01.bj" port: 18700 } } peers { permanent_uuid: "193181be29284287b25df2ee92938f0b" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st08.bj" port: 18700 } } I1031 23:05:01.801334 34067 leader_election.cc:216] T 6a813ca355f749b6b31a773aa0f2ac7a P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 0a8bbcbbb5a6459aa5258464755db3bb I1031 23:05:01.801369 34067 leader_election.cc:216] T 6a813ca355f749b6b31a773aa0f2ac7a P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 193181be29284287b25df2ee92938f0b W1031 23:05:01.801578 5816 leader_election.cc:278] T 6a813ca355f749b6b31a773aa0f2ac7a P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0a8bbcbbb5a6459aa5258464755db3bb: Illegal state: Tablet not RUNNING: NOT_STARTED W1031 23:05:01.801707 5815 leader_election.cc:278] T 6a813ca355f749b6b31a773aa0f2ac7a P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 193181be29284287b25df2ee92938f0b: Illegal state: Tablet not RUNNING: NOT_STARTED I1031 23:05:01.801728 5815 leader_election.cc:243] T 6a813ca355f749b6b31a773aa0f2ac7a P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. I1031 23:05:01.802119 34750 raft_consensus.cc:2162] T 6a813ca355f749b6b31a773aa0f2ac7a P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 12.243s I1031 23:05:01.802140 34750 raft_consensus.cc:1908] T 6a813ca355f749b6b31a773aa0f2ac7a P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority W1031 23:05:01.843325 5816 consensus_peers.cc:357] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 0a8bbcbbb5a6459aa5258464755db3bb (c3-kudu-tst-st02.bj:18700): Couldn't send request to peer 0a8bbcbbb5a6459aa5258464755db3bb for tablet 648eb376255a4dd4bb5818262efcc813. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 138 times. W1031 23:05:02.076213 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 78 times. W1031 23:05:02.091593 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 138 times. W1031 23:05:02.343423 5816 consensus_peers.cc:357] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 0a8bbcbbb5a6459aa5258464755db3bb (c3-kudu-tst-st02.bj:18700): Couldn't send request to peer 0a8bbcbbb5a6459aa5258464755db3bb for tablet 648eb376255a4dd4bb5818262efcc813. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 139 times. W1031 23:05:02.576308 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 79 times. W1031 23:05:02.591686 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 139 times. I1031 23:05:02.712393 34585 raft_consensus.cc:411] T 56155a2adc24450d89e338cc955bfa18 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I1031 23:05:02.712437 34585 raft_consensus.cc:2162] T 56155a2adc24450d89e338cc955bfa18 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 5.055s I1031 23:05:02.712502 34585 raft_consensus.cc:435] T 56155a2adc24450d89e338cc955bfa18 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625f49b9554549e6a5c3a92b76e576cf" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st10.bj" port: 18700 } } peers { permanent_uuid: "193181be29284287b25df2ee92938f0b" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st08.bj" port: 18700 } } peers { permanent_uuid: "0f53a0d3ef7e44ebb0365c800752d5bd" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st01.bj" port: 18700 } } I1031 23:05:02.713098 34585 leader_election.cc:216] T 56155a2adc24450d89e338cc955bfa18 P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 625f49b9554549e6a5c3a92b76e576cf I1031 23:05:02.713124 34585 leader_election.cc:216] T 56155a2adc24450d89e338cc955bfa18 P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 193181be29284287b25df2ee92938f0b W1031 23:05:02.713369 5814 leader_election.cc:278] T 56155a2adc24450d89e338cc955bfa18 P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 625f49b9554549e6a5c3a92b76e576cf: Illegal state: Tablet not RUNNING: NOT_STARTED W1031 23:05:02.713449 5815 leader_election.cc:278] T 56155a2adc24450d89e338cc955bfa18 P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 193181be29284287b25df2ee92938f0b: Illegal state: Tablet not RUNNING: NOT_STARTED I1031 23:05:02.713484 5815 leader_election.cc:243] T 56155a2adc24450d89e338cc955bfa18 P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. I1031 23:05:02.714265 34753 raft_consensus.cc:2162] T 56155a2adc24450d89e338cc955bfa18 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 10.629s I1031 23:05:02.714287 34753 raft_consensus.cc:1908] T 56155a2adc24450d89e338cc955bfa18 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority W1031 23:05:02.843487 5816 consensus_peers.cc:357] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 0a8bbcbbb5a6459aa5258464755db3bb (c3-kudu-tst-st02.bj:18700): Couldn't send request to peer 0a8bbcbbb5a6459aa5258464755db3bb for tablet 648eb376255a4dd4bb5818262efcc813. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 140 times. W1031 23:05:03.076401 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 80 times. W1031 23:05:03.091703 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 140 times. W1031 23:05:03.343608 5816 consensus_peers.cc:357] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 0a8bbcbbb5a6459aa5258464755db3bb (c3-kudu-tst-st02.bj:18700): Couldn't send request to peer 0a8bbcbbb5a6459aa5258464755db3bb for tablet 648eb376255a4dd4bb5818262efcc813. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 141 times. W1031 23:05:03.576514 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 81 times. W1031 23:05:03.591871 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 141 times. W1031 23:05:03.843646 5816 consensus_peers.cc:357] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 0a8bbcbbb5a6459aa5258464755db3bb (c3-kudu-tst-st02.bj:18700): Couldn't send request to peer 0a8bbcbbb5a6459aa5258464755db3bb for tablet 648eb376255a4dd4bb5818262efcc813. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 142 times. I1031 23:05:03.942723 34574 raft_consensus.cc:411] T bff7dab34ccd4b42842077e82106f6fb P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I1031 23:05:03.942765 34574 raft_consensus.cc:2162] T bff7dab34ccd4b42842077e82106f6fb P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 2.395s I1031 23:05:03.942821 34574 raft_consensus.cc:435] T bff7dab34ccd4b42842077e82106f6fb P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f53a0d3ef7e44ebb0365c800752d5bd" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st01.bj" port: 18700 } } peers { permanent_uuid: "23f962e4a1744381ad5fa0d2d8b10241" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st07.bj" port: 18700 } } peers { permanent_uuid: "193181be29284287b25df2ee92938f0b" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st08.bj" port: 18700 } } I1031 23:05:03.943449 34574 leader_election.cc:216] T bff7dab34ccd4b42842077e82106f6fb P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 23f962e4a1744381ad5fa0d2d8b10241 I1031 23:05:03.943477 34574 leader_election.cc:216] T bff7dab34ccd4b42842077e82106f6fb P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 193181be29284287b25df2ee92938f0b W1031 23:05:03.943732 5816 leader_election.cc:278] T bff7dab34ccd4b42842077e82106f6fb P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 23f962e4a1744381ad5fa0d2d8b10241: Illegal state: Tablet not RUNNING: NOT_STARTED W1031 23:05:03.943773 5815 leader_election.cc:278] T bff7dab34ccd4b42842077e82106f6fb P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 193181be29284287b25df2ee92938f0b: Illegal state: Tablet not RUNNING: NOT_STARTED I1031 23:05:03.943795 5815 leader_election.cc:243] T bff7dab34ccd4b42842077e82106f6fb P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. I1031 23:05:03.944046 34755 raft_consensus.cc:2162] T bff7dab34ccd4b42842077e82106f6fb P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 7.819s I1031 23:05:03.944064 34755 raft_consensus.cc:1908] T bff7dab34ccd4b42842077e82106f6fb P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority W1031 23:05:04.076544 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 82 times. W1031 23:05:04.091883 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 142 times. W1031 23:05:04.343734 5816 consensus_peers.cc:357] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 0a8bbcbbb5a6459aa5258464755db3bb (c3-kudu-tst-st02.bj:18700): Couldn't send request to peer 0a8bbcbbb5a6459aa5258464755db3bb for tablet 648eb376255a4dd4bb5818262efcc813. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 143 times. W1031 23:05:04.576608 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 83 times. W1031 23:05:04.591943 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 143 times. W1031 23:05:04.843823 5816 consensus_peers.cc:357] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 0a8bbcbbb5a6459aa5258464755db3bb (c3-kudu-tst-st02.bj:18700): Couldn't send request to peer 0a8bbcbbb5a6459aa5258464755db3bb for tablet 648eb376255a4dd4bb5818262efcc813. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 144 times. W1031 23:05:05.076629 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 84 times. W1031 23:05:05.092061 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 144 times. W1031 23:05:05.343910 5816 consensus_peers.cc:357] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 0a8bbcbbb5a6459aa5258464755db3bb (c3-kudu-tst-st02.bj:18700): Couldn't send request to peer 0a8bbcbbb5a6459aa5258464755db3bb for tablet 648eb376255a4dd4bb5818262efcc813. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 145 times. W1031 23:05:05.576660 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 85 times. W1031 23:05:05.592156 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 145 times. W1031 23:05:05.843924 5816 consensus_peers.cc:357] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 0a8bbcbbb5a6459aa5258464755db3bb (c3-kudu-tst-st02.bj:18700): Couldn't send request to peer 0a8bbcbbb5a6459aa5258464755db3bb for tablet 648eb376255a4dd4bb5818262efcc813. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 146 times. W1031 23:05:06.076763 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 86 times. W1031 23:05:06.092216 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 146 times. W1031 23:05:06.343997 5816 consensus_peers.cc:357] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 0a8bbcbbb5a6459aa5258464755db3bb (c3-kudu-tst-st02.bj:18700): Couldn't send request to peer 0a8bbcbbb5a6459aa5258464755db3bb for tablet 648eb376255a4dd4bb5818262efcc813. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 147 times. W1031 23:05:06.576823 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 87 times. W1031 23:05:06.592298 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 147 times. W1031 23:05:06.844055 5816 consensus_peers.cc:357] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 0a8bbcbbb5a6459aa5258464755db3bb (c3-kudu-tst-st02.bj:18700): Couldn't send request to peer 0a8bbcbbb5a6459aa5258464755db3bb for tablet 648eb376255a4dd4bb5818262efcc813. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 148 times. W1031 23:05:07.076932 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 88 times. W1031 23:05:07.092519 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 148 times. W1031 23:05:07.344132 5816 consensus_peers.cc:357] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 0a8bbcbbb5a6459aa5258464755db3bb (c3-kudu-tst-st02.bj:18700): Couldn't send request to peer 0a8bbcbbb5a6459aa5258464755db3bb for tablet 648eb376255a4dd4bb5818262efcc813. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 149 times. W1031 23:05:07.577041 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 89 times. W1031 23:05:07.592711 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 149 times. W1031 23:05:07.844279 5816 consensus_peers.cc:357] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 0a8bbcbbb5a6459aa5258464755db3bb (c3-kudu-tst-st02.bj:18700): Couldn't send request to peer 0a8bbcbbb5a6459aa5258464755db3bb for tablet 648eb376255a4dd4bb5818262efcc813. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 150 times. W1031 23:05:08.077066 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 90 times. W1031 23:05:08.092707 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 150 times. W1031 23:05:08.344357 5816 consensus_peers.cc:357] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 0a8bbcbbb5a6459aa5258464755db3bb (c3-kudu-tst-st02.bj:18700): Couldn't send request to peer 0a8bbcbbb5a6459aa5258464755db3bb for tablet 648eb376255a4dd4bb5818262efcc813. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 151 times. W1031 23:05:08.577152 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 91 times. W1031 23:05:08.592792 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 151 times. I1031 23:05:08.718745 34590 raft_consensus.cc:411] T 117cd74bcdab4942901acb0005cdd702 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I1031 23:05:08.718781 34590 raft_consensus.cc:2162] T 117cd74bcdab4942901acb0005cdd702 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.633s I1031 23:05:08.718833 34590 raft_consensus.cc:435] T 117cd74bcdab4942901acb0005cdd702 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625f49b9554549e6a5c3a92b76e576cf" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st10.bj" port: 18700 } } peers { permanent_uuid: "0a8bbcbbb5a6459aa5258464755db3bb" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st02.bj" port: 18700 } } peers { permanent_uuid: "0f53a0d3ef7e44ebb0365c800752d5bd" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st01.bj" port: 18700 } } I1031 23:05:08.719476 34590 leader_election.cc:216] T 117cd74bcdab4942901acb0005cdd702 P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 625f49b9554549e6a5c3a92b76e576cf I1031 23:05:08.719502 34590 leader_election.cc:216] T 117cd74bcdab4942901acb0005cdd702 P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 0a8bbcbbb5a6459aa5258464755db3bb W1031 23:05:08.719769 5816 leader_election.cc:278] T 117cd74bcdab4942901acb0005cdd702 P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0a8bbcbbb5a6459aa5258464755db3bb: Illegal state: Tablet not RUNNING: NOT_STARTED W1031 23:05:08.719804 5814 leader_election.cc:278] T 117cd74bcdab4942901acb0005cdd702 P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 625f49b9554549e6a5c3a92b76e576cf: Illegal state: Tablet not RUNNING: NOT_STARTED I1031 23:05:08.719825 5814 leader_election.cc:243] T 117cd74bcdab4942901acb0005cdd702 P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. I1031 23:05:08.720654 34780 raft_consensus.cc:2162] T 117cd74bcdab4942901acb0005cdd702 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 2.944s I1031 23:05:08.720676 34780 raft_consensus.cc:1908] T 117cd74bcdab4942901acb0005cdd702 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority W1031 23:05:08.844357 5816 consensus_peers.cc:357] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 0a8bbcbbb5a6459aa5258464755db3bb (c3-kudu-tst-st02.bj:18700): Couldn't send request to peer 0a8bbcbbb5a6459aa5258464755db3bb for tablet 648eb376255a4dd4bb5818262efcc813. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 152 times. I1031 23:05:09.036890 5857 tablet_service.cc:691] Processing DeleteTablet for tablet 117cd74bcdab4942901acb0005cdd702 with delete_type TABLET_DATA_DELETED (Replaced by acb401e40e7049fbbfdbeb40474353d9 at 2017-10-31 23:04:52 CST) from {username='work'} at 10.108.187.10:53997 W1031 23:05:09.077136 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 92 times. W1031 23:05:09.092782 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 152 times. I1031 23:05:09.129900 5857 tablet_service.cc:691] Processing DeleteTablet for tablet 6a813ca355f749b6b31a773aa0f2ac7a with delete_type TABLET_DATA_DELETED (Replaced by 582c1281bf5147b2bd40c9a950a1152f at 2017-10-31 23:04:22 CST) from {username='work'} at 10.108.187.10:53997 I1031 23:05:09.137367 5857 tablet_service.cc:691] Processing DeleteTablet for tablet bff7dab34ccd4b42842077e82106f6fb with delete_type TABLET_DATA_DELETED (Replaced by 005a48fb466c4a52896842dd00124755 at 2017-10-31 23:04:52 CST) from {username='work'} at 10.108.187.10:53997 I1031 23:05:09.174872 5857 tablet_service.cc:691] Processing DeleteTablet for tablet 56155a2adc24450d89e338cc955bfa18 with delete_type TABLET_DATA_DELETED (Replaced by 80ba7f33ec864c5f96c61932edb5cdc4 at 2017-10-31 23:04:52 CST) from {username='work'} at 10.108.187.10:53997 W1031 23:05:09.344485 5816 consensus_peers.cc:357] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 0a8bbcbbb5a6459aa5258464755db3bb (c3-kudu-tst-st02.bj:18700): Couldn't send request to peer 0a8bbcbbb5a6459aa5258464755db3bb for tablet 648eb376255a4dd4bb5818262efcc813. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 153 times. W1031 23:05:09.577312 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 93 times. W1031 23:05:09.592949 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 153 times. W1031 23:05:09.844470 5816 consensus_peers.cc:357] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 0a8bbcbbb5a6459aa5258464755db3bb (c3-kudu-tst-st02.bj:18700): Couldn't send request to peer 0a8bbcbbb5a6459aa5258464755db3bb for tablet 648eb376255a4dd4bb5818262efcc813. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 154 times. W1031 23:05:10.077358 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 94 times. W1031 23:05:10.093086 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 154 times. W1031 23:05:10.344539 5816 consensus_peers.cc:357] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 0a8bbcbbb5a6459aa5258464755db3bb (c3-kudu-tst-st02.bj:18700): Couldn't send request to peer 0a8bbcbbb5a6459aa5258464755db3bb for tablet 648eb376255a4dd4bb5818262efcc813. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 155 times. W1031 23:05:10.577549 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 95 times. W1031 23:05:10.593204 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 155 times. W1031 23:05:10.844609 5816 consensus_peers.cc:357] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 0a8bbcbbb5a6459aa5258464755db3bb (c3-kudu-tst-st02.bj:18700): Couldn't send request to peer 0a8bbcbbb5a6459aa5258464755db3bb for tablet 648eb376255a4dd4bb5818262efcc813. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 156 times. W1031 23:05:11.077517 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 96 times. W1031 23:05:11.093237 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 156 times. W1031 23:05:11.344637 5816 consensus_peers.cc:357] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 0a8bbcbbb5a6459aa5258464755db3bb (c3-kudu-tst-st02.bj:18700): Couldn't send request to peer 0a8bbcbbb5a6459aa5258464755db3bb for tablet 648eb376255a4dd4bb5818262efcc813. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 157 times. W1031 23:05:11.577648 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 97 times. W1031 23:05:11.593396 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 157 times. W1031 23:05:11.844718 5816 consensus_peers.cc:357] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 0a8bbcbbb5a6459aa5258464755db3bb (c3-kudu-tst-st02.bj:18700): Couldn't send request to peer 0a8bbcbbb5a6459aa5258464755db3bb for tablet 648eb376255a4dd4bb5818262efcc813. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 158 times. W1031 23:05:12.077622 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 98 times. W1031 23:05:12.093344 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 158 times. W1031 23:05:12.344898 5816 consensus_peers.cc:357] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 0a8bbcbbb5a6459aa5258464755db3bb (c3-kudu-tst-st02.bj:18700): Couldn't send request to peer 0a8bbcbbb5a6459aa5258464755db3bb for tablet 648eb376255a4dd4bb5818262efcc813. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 159 times. W1031 23:05:12.577759 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 99 times. W1031 23:05:12.593430 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 159 times. W1031 23:05:12.844874 5816 consensus_peers.cc:357] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 0a8bbcbbb5a6459aa5258464755db3bb (c3-kudu-tst-st02.bj:18700): Couldn't send request to peer 0a8bbcbbb5a6459aa5258464755db3bb for tablet 648eb376255a4dd4bb5818262efcc813. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 160 times. W1031 23:05:13.077855 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 100 times. W1031 23:05:13.093542 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 160 times. W1031 23:05:13.344997 5816 consensus_peers.cc:357] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 0a8bbcbbb5a6459aa5258464755db3bb (c3-kudu-tst-st02.bj:18700): Couldn't send request to peer 0a8bbcbbb5a6459aa5258464755db3bb for tablet 648eb376255a4dd4bb5818262efcc813. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 161 times. W1031 23:05:13.578012 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 101 times. W1031 23:05:13.593713 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 161 times. I1031 23:05:13.706460 34574 raft_consensus.cc:411] T bff7dab34ccd4b42842077e82106f6fb P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I1031 23:05:13.706496 34574 raft_consensus.cc:2162] T bff7dab34ccd4b42842077e82106f6fb P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 5.301s I1031 23:05:13.706547 34574 raft_consensus.cc:435] T bff7dab34ccd4b42842077e82106f6fb P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f53a0d3ef7e44ebb0365c800752d5bd" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st01.bj" port: 18700 } } peers { permanent_uuid: "23f962e4a1744381ad5fa0d2d8b10241" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st07.bj" port: 18700 } } peers { permanent_uuid: "193181be29284287b25df2ee92938f0b" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st08.bj" port: 18700 } } I1031 23:05:13.707165 34574 leader_election.cc:216] T bff7dab34ccd4b42842077e82106f6fb P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 23f962e4a1744381ad5fa0d2d8b10241 I1031 23:05:13.707191 34574 leader_election.cc:216] T bff7dab34ccd4b42842077e82106f6fb P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 193181be29284287b25df2ee92938f0b W1031 23:05:13.707458 5816 leader_election.cc:278] T bff7dab34ccd4b42842077e82106f6fb P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 23f962e4a1744381ad5fa0d2d8b10241: Illegal state: Tablet not RUNNING: NOT_STARTED W1031 23:05:13.707562 5815 leader_election.cc:278] T bff7dab34ccd4b42842077e82106f6fb P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 193181be29284287b25df2ee92938f0b: Illegal state: Tablet not RUNNING: NOT_STARTED I1031 23:05:13.707587 5815 leader_election.cc:243] T bff7dab34ccd4b42842077e82106f6fb P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. I1031 23:05:13.707849 34811 raft_consensus.cc:2162] T bff7dab34ccd4b42842077e82106f6fb P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 19.158s I1031 23:05:13.707870 34811 raft_consensus.cc:1908] T bff7dab34ccd4b42842077e82106f6fb P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority I1031 23:05:13.787355 34590 raft_consensus.cc:411] T 117cd74bcdab4942901acb0005cdd702 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I1031 23:05:13.787389 34590 raft_consensus.cc:2162] T 117cd74bcdab4942901acb0005cdd702 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 2.138s I1031 23:05:13.787423 34590 raft_consensus.cc:435] T 117cd74bcdab4942901acb0005cdd702 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625f49b9554549e6a5c3a92b76e576cf" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st10.bj" port: 18700 } } peers { permanent_uuid: "0a8bbcbbb5a6459aa5258464755db3bb" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st02.bj" port: 18700 } } peers { permanent_uuid: "0f53a0d3ef7e44ebb0365c800752d5bd" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st01.bj" port: 18700 } } I1031 23:05:13.788030 34590 leader_election.cc:216] T 117cd74bcdab4942901acb0005cdd702 P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 625f49b9554549e6a5c3a92b76e576cf I1031 23:05:13.788071 34590 leader_election.cc:216] T 117cd74bcdab4942901acb0005cdd702 P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 0a8bbcbbb5a6459aa5258464755db3bb W1031 23:05:13.788295 5816 leader_election.cc:278] T 117cd74bcdab4942901acb0005cdd702 P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0a8bbcbbb5a6459aa5258464755db3bb: Illegal state: Tablet not RUNNING: NOT_STARTED W1031 23:05:13.788326 5814 leader_election.cc:278] T 117cd74bcdab4942901acb0005cdd702 P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 625f49b9554549e6a5c3a92b76e576cf: Illegal state: Tablet not RUNNING: NOT_STARTED I1031 23:05:13.788345 5814 leader_election.cc:243] T 117cd74bcdab4942901acb0005cdd702 P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. I1031 23:05:13.788528 34812 raft_consensus.cc:2162] T 117cd74bcdab4942901acb0005cdd702 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 6.938s I1031 23:05:13.788549 34812 raft_consensus.cc:1908] T 117cd74bcdab4942901acb0005cdd702 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority W1031 23:05:13.844933 5816 consensus_peers.cc:357] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 0a8bbcbbb5a6459aa5258464755db3bb (c3-kudu-tst-st02.bj:18700): Couldn't send request to peer 0a8bbcbbb5a6459aa5258464755db3bb for tablet 648eb376255a4dd4bb5818262efcc813. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 162 times. W1031 23:05:14.077987 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 102 times. W1031 23:05:14.093796 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 162 times. W1031 23:05:14.345084 5816 consensus_peers.cc:357] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 0a8bbcbbb5a6459aa5258464755db3bb (c3-kudu-tst-st02.bj:18700): Couldn't send request to peer 0a8bbcbbb5a6459aa5258464755db3bb for tablet 648eb376255a4dd4bb5818262efcc813. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 163 times. W1031 23:05:14.578070 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 103 times. W1031 23:05:14.593858 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 163 times. W1031 23:05:14.845191 5816 consensus_peers.cc:357] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 0a8bbcbbb5a6459aa5258464755db3bb (c3-kudu-tst-st02.bj:18700): Couldn't send request to peer 0a8bbcbbb5a6459aa5258464755db3bb for tablet 648eb376255a4dd4bb5818262efcc813. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 164 times. W1031 23:05:15.078096 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 104 times. W1031 23:05:15.093822 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 164 times. W1031 23:05:15.345278 5816 consensus_peers.cc:357] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 0a8bbcbbb5a6459aa5258464755db3bb (c3-kudu-tst-st02.bj:18700): Couldn't send request to peer 0a8bbcbbb5a6459aa5258464755db3bb for tablet 648eb376255a4dd4bb5818262efcc813. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 165 times. W1031 23:05:15.578238 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 105 times. W1031 23:05:15.593951 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 165 times. I1031 23:05:15.840790 34585 raft_consensus.cc:411] T 56155a2adc24450d89e338cc955bfa18 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I1031 23:05:15.840843 34585 raft_consensus.cc:2162] T 56155a2adc24450d89e338cc955bfa18 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 8.431s I1031 23:05:15.840903 34585 raft_consensus.cc:435] T 56155a2adc24450d89e338cc955bfa18 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625f49b9554549e6a5c3a92b76e576cf" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st10.bj" port: 18700 } } peers { permanent_uuid: "193181be29284287b25df2ee92938f0b" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st08.bj" port: 18700 } } peers { permanent_uuid: "0f53a0d3ef7e44ebb0365c800752d5bd" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st01.bj" port: 18700 } } I1031 23:05:15.841593 34585 leader_election.cc:216] T 56155a2adc24450d89e338cc955bfa18 P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 625f49b9554549e6a5c3a92b76e576cf I1031 23:05:15.841622 34585 leader_election.cc:216] T 56155a2adc24450d89e338cc955bfa18 P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 193181be29284287b25df2ee92938f0b W1031 23:05:15.841893 5815 leader_election.cc:278] T 56155a2adc24450d89e338cc955bfa18 P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 193181be29284287b25df2ee92938f0b: Illegal state: Tablet not RUNNING: NOT_STARTED W1031 23:05:15.841980 5814 leader_election.cc:278] T 56155a2adc24450d89e338cc955bfa18 P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 625f49b9554549e6a5c3a92b76e576cf: Illegal state: Tablet not RUNNING: NOT_STARTED I1031 23:05:15.842020 5814 leader_election.cc:243] T 56155a2adc24450d89e338cc955bfa18 P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. I1031 23:05:15.842803 35053 raft_consensus.cc:2162] T 56155a2adc24450d89e338cc955bfa18 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 2.235s I1031 23:05:15.842825 35053 raft_consensus.cc:1908] T 56155a2adc24450d89e338cc955bfa18 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority W1031 23:05:15.845242 5816 consensus_peers.cc:357] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 0a8bbcbbb5a6459aa5258464755db3bb (c3-kudu-tst-st02.bj:18700): Couldn't send request to peer 0a8bbcbbb5a6459aa5258464755db3bb for tablet 648eb376255a4dd4bb5818262efcc813. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 166 times. I1031 23:05:15.957181 34067 raft_consensus.cc:411] T 6a813ca355f749b6b31a773aa0f2ac7a P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I1031 23:05:15.957234 34067 raft_consensus.cc:2162] T 6a813ca355f749b6b31a773aa0f2ac7a P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 3.881s I1031 23:05:15.957291 34067 raft_consensus.cc:435] T 6a813ca355f749b6b31a773aa0f2ac7a P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a8bbcbbb5a6459aa5258464755db3bb" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st02.bj" port: 18700 } } peers { permanent_uuid: "0f53a0d3ef7e44ebb0365c800752d5bd" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st01.bj" port: 18700 } } peers { permanent_uuid: "193181be29284287b25df2ee92938f0b" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st08.bj" port: 18700 } } I1031 23:05:15.958003 34067 leader_election.cc:216] T 6a813ca355f749b6b31a773aa0f2ac7a P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 0a8bbcbbb5a6459aa5258464755db3bb I1031 23:05:15.958036 34067 leader_election.cc:216] T 6a813ca355f749b6b31a773aa0f2ac7a P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 193181be29284287b25df2ee92938f0b W1031 23:05:15.958230 5816 leader_election.cc:278] T 6a813ca355f749b6b31a773aa0f2ac7a P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0a8bbcbbb5a6459aa5258464755db3bb: Illegal state: Tablet not RUNNING: NOT_STARTED W1031 23:05:15.958413 5815 leader_election.cc:278] T 6a813ca355f749b6b31a773aa0f2ac7a P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 193181be29284287b25df2ee92938f0b: Illegal state: Tablet not RUNNING: NOT_STARTED I1031 23:05:15.958446 5815 leader_election.cc:243] T 6a813ca355f749b6b31a773aa0f2ac7a P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. I1031 23:05:15.958999 35054 raft_consensus.cc:2162] T 6a813ca355f749b6b31a773aa0f2ac7a P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 16.493s I1031 23:05:15.959022 35054 raft_consensus.cc:1908] T 6a813ca355f749b6b31a773aa0f2ac7a P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority W1031 23:05:16.078228 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 106 times. W1031 23:05:16.094071 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 166 times. W1031 23:05:16.345407 5816 consensus_peers.cc:357] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 0a8bbcbbb5a6459aa5258464755db3bb (c3-kudu-tst-st02.bj:18700): Couldn't send request to peer 0a8bbcbbb5a6459aa5258464755db3bb for tablet 648eb376255a4dd4bb5818262efcc813. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 167 times. W1031 23:05:16.578378 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 107 times. W1031 23:05:16.594076 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 167 times. W1031 23:05:16.845386 5816 consensus_peers.cc:357] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 0a8bbcbbb5a6459aa5258464755db3bb (c3-kudu-tst-st02.bj:18700): Couldn't send request to peer 0a8bbcbbb5a6459aa5258464755db3bb for tablet 648eb376255a4dd4bb5818262efcc813. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 168 times. W1031 23:05:17.078451 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 108 times. W1031 23:05:17.094187 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 168 times. W1031 23:05:17.345538 5816 consensus_peers.cc:357] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 0a8bbcbbb5a6459aa5258464755db3bb (c3-kudu-tst-st02.bj:18700): Couldn't send request to peer 0a8bbcbbb5a6459aa5258464755db3bb for tablet 648eb376255a4dd4bb5818262efcc813. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 169 times. W1031 23:05:17.578507 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 109 times. W1031 23:05:17.594271 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 169 times. W1031 23:05:17.845573 5816 consensus_peers.cc:357] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 0a8bbcbbb5a6459aa5258464755db3bb (c3-kudu-tst-st02.bj:18700): Couldn't send request to peer 0a8bbcbbb5a6459aa5258464755db3bb for tablet 648eb376255a4dd4bb5818262efcc813. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 170 times. W1031 23:05:18.078505 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 110 times. W1031 23:05:18.094415 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 170 times. W1031 23:05:18.345656 5816 consensus_peers.cc:357] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 0a8bbcbbb5a6459aa5258464755db3bb (c3-kudu-tst-st02.bj:18700): Couldn't send request to peer 0a8bbcbbb5a6459aa5258464755db3bb for tablet 648eb376255a4dd4bb5818262efcc813. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 171 times. W1031 23:05:18.578641 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 111 times. W1031 23:05:18.594410 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 171 times. W1031 23:05:18.845726 5816 consensus_peers.cc:357] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 0a8bbcbbb5a6459aa5258464755db3bb (c3-kudu-tst-st02.bj:18700): Couldn't send request to peer 0a8bbcbbb5a6459aa5258464755db3bb for tablet 648eb376255a4dd4bb5818262efcc813. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 172 times. W1031 23:05:19.078716 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 112 times. W1031 23:05:19.094496 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 172 times. W1031 23:05:19.345814 5816 consensus_peers.cc:357] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 0a8bbcbbb5a6459aa5258464755db3bb (c3-kudu-tst-st02.bj:18700): Couldn't send request to peer 0a8bbcbbb5a6459aa5258464755db3bb for tablet 648eb376255a4dd4bb5818262efcc813. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 173 times. W1031 23:05:19.578891 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 113 times. W1031 23:05:19.594633 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 173 times. I1031 23:05:19.774404 34585 raft_consensus.cc:411] T 56155a2adc24450d89e338cc955bfa18 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I1031 23:05:19.774442 34585 raft_consensus.cc:2162] T 56155a2adc24450d89e338cc955bfa18 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 15.280s I1031 23:05:19.774493 34585 raft_consensus.cc:435] T 56155a2adc24450d89e338cc955bfa18 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625f49b9554549e6a5c3a92b76e576cf" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st10.bj" port: 18700 } } peers { permanent_uuid: "193181be29284287b25df2ee92938f0b" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st08.bj" port: 18700 } } peers { permanent_uuid: "0f53a0d3ef7e44ebb0365c800752d5bd" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st01.bj" port: 18700 } } I1031 23:05:19.774996 34585 leader_election.cc:216] T 56155a2adc24450d89e338cc955bfa18 P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 625f49b9554549e6a5c3a92b76e576cf I1031 23:05:19.775022 34585 leader_election.cc:216] T 56155a2adc24450d89e338cc955bfa18 P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 193181be29284287b25df2ee92938f0b W1031 23:05:19.775333 5814 leader_election.cc:278] T 56155a2adc24450d89e338cc955bfa18 P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 625f49b9554549e6a5c3a92b76e576cf: Illegal state: Tablet not RUNNING: NOT_STARTED W1031 23:05:19.775382 5815 leader_election.cc:278] T 56155a2adc24450d89e338cc955bfa18 P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 193181be29284287b25df2ee92938f0b: Illegal state: Tablet not RUNNING: NOT_STARTED I1031 23:05:19.775418 5815 leader_election.cc:243] T 56155a2adc24450d89e338cc955bfa18 P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. I1031 23:05:19.775748 35060 raft_consensus.cc:2162] T 56155a2adc24450d89e338cc955bfa18 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 9.480s I1031 23:05:19.775770 35060 raft_consensus.cc:1908] T 56155a2adc24450d89e338cc955bfa18 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority W1031 23:05:19.845845 5816 consensus_peers.cc:357] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 0a8bbcbbb5a6459aa5258464755db3bb (c3-kudu-tst-st02.bj:18700): Couldn't send request to peer 0a8bbcbbb5a6459aa5258464755db3bb for tablet 648eb376255a4dd4bb5818262efcc813. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 174 times. W1031 23:05:20.078856 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 114 times. W1031 23:05:20.094586 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 174 times. W1031 23:05:20.345949 5816 consensus_peers.cc:357] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 0a8bbcbbb5a6459aa5258464755db3bb (c3-kudu-tst-st02.bj:18700): Couldn't send request to peer 0a8bbcbbb5a6459aa5258464755db3bb for tablet 648eb376255a4dd4bb5818262efcc813. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 175 times. W1031 23:05:20.579000 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 115 times. W1031 23:05:20.594697 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 175 times. W1031 23:05:20.846097 5816 consensus_peers.cc:357] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 0a8bbcbbb5a6459aa5258464755db3bb (c3-kudu-tst-st02.bj:18700): Couldn't send request to peer 0a8bbcbbb5a6459aa5258464755db3bb for tablet 648eb376255a4dd4bb5818262efcc813. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 176 times. W1031 23:05:21.079087 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 116 times. W1031 23:05:21.094872 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 176 times. W1031 23:05:21.346123 5816 consensus_peers.cc:357] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 0a8bbcbbb5a6459aa5258464755db3bb (c3-kudu-tst-st02.bj:18700): Couldn't send request to peer 0a8bbcbbb5a6459aa5258464755db3bb for tablet 648eb376255a4dd4bb5818262efcc813. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 177 times. W1031 23:05:21.579227 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 117 times. W1031 23:05:21.594925 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 177 times. W1031 23:05:21.846138 5816 consensus_peers.cc:357] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 0a8bbcbbb5a6459aa5258464755db3bb (c3-kudu-tst-st02.bj:18700): Couldn't send request to peer 0a8bbcbbb5a6459aa5258464755db3bb for tablet 648eb376255a4dd4bb5818262efcc813. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 178 times. W1031 23:05:22.079200 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 118 times. W1031 23:05:22.095006 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 178 times. W1031 23:05:22.346243 5816 consensus_peers.cc:357] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 0a8bbcbbb5a6459aa5258464755db3bb (c3-kudu-tst-st02.bj:18700): Couldn't send request to peer 0a8bbcbbb5a6459aa5258464755db3bb for tablet 648eb376255a4dd4bb5818262efcc813. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 179 times. I1031 23:05:22.373450 34590 raft_consensus.cc:411] T 117cd74bcdab4942901acb0005cdd702 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I1031 23:05:22.373517 34590 raft_consensus.cc:2162] T 117cd74bcdab4942901acb0005cdd702 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 14.600s I1031 23:05:22.373581 34590 raft_consensus.cc:435] T 117cd74bcdab4942901acb0005cdd702 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625f49b9554549e6a5c3a92b76e576cf" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st10.bj" port: 18700 } } peers { permanent_uuid: "0a8bbcbbb5a6459aa5258464755db3bb" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st02.bj" port: 18700 } } peers { permanent_uuid: "0f53a0d3ef7e44ebb0365c800752d5bd" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st01.bj" port: 18700 } } I1031 23:05:22.374210 34590 leader_election.cc:216] T 117cd74bcdab4942901acb0005cdd702 P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 625f49b9554549e6a5c3a92b76e576cf I1031 23:05:22.374245 34590 leader_election.cc:216] T 117cd74bcdab4942901acb0005cdd702 P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 0a8bbcbbb5a6459aa5258464755db3bb W1031 23:05:22.374444 5814 leader_election.cc:278] T 117cd74bcdab4942901acb0005cdd702 P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 625f49b9554549e6a5c3a92b76e576cf: Illegal state: Tablet not RUNNING: NOT_STARTED W1031 23:05:22.374478 5816 leader_election.cc:278] T 117cd74bcdab4942901acb0005cdd702 P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0a8bbcbbb5a6459aa5258464755db3bb: Illegal state: Tablet not RUNNING: NOT_STARTED I1031 23:05:22.374500 5816 leader_election.cc:243] T 117cd74bcdab4942901acb0005cdd702 P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. I1031 23:05:22.375192 35065 raft_consensus.cc:2162] T 117cd74bcdab4942901acb0005cdd702 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 10.081s I1031 23:05:22.375214 35065 raft_consensus.cc:1908] T 117cd74bcdab4942901acb0005cdd702 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority I1031 23:05:22.526315 5857 tablet_service.cc:650] Processing CreateTablet for tablet 012b21c0f95e4dd594076018fd6fdb7e (table=default.timeout_tst [id=cdcb3e60c1314776843d856365535d49]), partition=HASH (id) PARTITION 4, RANGE (date) PARTITION 20171025 <= VALUES < 20171026 I1031 23:05:22.526378 5866 tablet_service.cc:650] Processing CreateTablet for tablet 95f15519e52a48edab3b765ae3a8e93d (table=default.timeout_tst [id=cdcb3e60c1314776843d856365535d49]), partition=HASH (id) PARTITION 4, RANGE (date) PARTITION 20171027 <= VALUES < 20171028 I1031 23:05:22.527453 5861 tablet_service.cc:650] Processing CreateTablet for tablet b4e182b626554f3597d5c733c80997e6 (table=default.timeout_tst [id=cdcb3e60c1314776843d856365535d49]), partition=HASH (id) PARTITION 4, RANGE (date) PARTITION 20171029 <= VALUES < 20171030 W1031 23:05:22.579232 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 119 times. I1031 23:05:22.581339 5866 ts_tablet_manager.cc:862] T 95f15519e52a48edab3b765ae3a8e93d P 0f53a0d3ef7e44ebb0365c800752d5bd: Registered tablet (data state: TABLET_DATA_READY) I1031 23:05:22.582216 35069 ts_tablet_manager.cc:723] T 95f15519e52a48edab3b765ae3a8e93d P 0f53a0d3ef7e44ebb0365c800752d5bd: Bootstrapping tablet I1031 23:05:22.582610 35069 tablet_bootstrap.cc:382] T 95f15519e52a48edab3b765ae3a8e93d P 0f53a0d3ef7e44ebb0365c800752d5bd: Bootstrap starting. I1031 23:05:22.582775 35069 tablet_bootstrap.cc:541] T 95f15519e52a48edab3b765ae3a8e93d P 0f53a0d3ef7e44ebb0365c800752d5bd: Time spent opening tablet: real 0.000s user 0.000s sys 0.000s I1031 23:05:22.582916 35069 tablet_bootstrap.cc:484] T 95f15519e52a48edab3b765ae3a8e93d P 0f53a0d3ef7e44ebb0365c800752d5bd: No blocks or log segments found. Creating new log. I1031 23:05:22.584532 35069 tablet_bootstrap.cc:382] T 95f15519e52a48edab3b765ae3a8e93d P 0f53a0d3ef7e44ebb0365c800752d5bd: No bootstrap required, opened a new log I1031 23:05:22.584560 35069 ts_tablet_manager.cc:728] T 95f15519e52a48edab3b765ae3a8e93d P 0f53a0d3ef7e44ebb0365c800752d5bd: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s I1031 23:05:22.585342 35069 raft_consensus.cc:288] T 95f15519e52a48edab3b765ae3a8e93d P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f53a0d3ef7e44ebb0365c800752d5bd" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st01.bj" port: 18700 } } peers { permanent_uuid: "d3eb30bd1bc34b95902fa61724239553" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st05.bj" port: 18700 } } peers { permanent_uuid: "193181be29284287b25df2ee92938f0b" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st08.bj" port: 18700 } } I1031 23:05:22.585366 35069 raft_consensus.cc:319] T 95f15519e52a48edab3b765ae3a8e93d P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1031 23:05:22.585377 35069 raft_consensus.cc:548] T 95f15519e52a48edab3b765ae3a8e93d P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f53a0d3ef7e44ebb0365c800752d5bd, State: 1, Role: FOLLOWER I1031 23:05:22.585394 35069 consensus_queue.cc:176] T 95f15519e52a48edab3b765ae3a8e93d P 0f53a0d3ef7e44ebb0365c800752d5bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER W1031 23:05:22.594977 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 179 times. I1031 23:05:22.606559 5861 ts_tablet_manager.cc:862] T b4e182b626554f3597d5c733c80997e6 P 0f53a0d3ef7e44ebb0365c800752d5bd: Registered tablet (data state: TABLET_DATA_READY) I1031 23:05:22.607147 35074 ts_tablet_manager.cc:723] T b4e182b626554f3597d5c733c80997e6 P 0f53a0d3ef7e44ebb0365c800752d5bd: Bootstrapping tablet I1031 23:05:22.607594 35074 tablet_bootstrap.cc:382] T b4e182b626554f3597d5c733c80997e6 P 0f53a0d3ef7e44ebb0365c800752d5bd: Bootstrap starting. I1031 23:05:22.607713 35074 tablet_bootstrap.cc:541] T b4e182b626554f3597d5c733c80997e6 P 0f53a0d3ef7e44ebb0365c800752d5bd: Time spent opening tablet: real 0.000s user 0.000s sys 0.000s I1031 23:05:22.607820 35074 tablet_bootstrap.cc:484] T b4e182b626554f3597d5c733c80997e6 P 0f53a0d3ef7e44ebb0365c800752d5bd: No blocks or log segments found. Creating new log. I1031 23:05:22.609000 35074 tablet_bootstrap.cc:382] T b4e182b626554f3597d5c733c80997e6 P 0f53a0d3ef7e44ebb0365c800752d5bd: No bootstrap required, opened a new log I1031 23:05:22.609027 35074 ts_tablet_manager.cc:728] T b4e182b626554f3597d5c733c80997e6 P 0f53a0d3ef7e44ebb0365c800752d5bd: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s I1031 23:05:22.609829 35074 raft_consensus.cc:288] T b4e182b626554f3597d5c733c80997e6 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3eb30bd1bc34b95902fa61724239553" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st05.bj" port: 18700 } } peers { permanent_uuid: "193181be29284287b25df2ee92938f0b" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st08.bj" port: 18700 } } peers { permanent_uuid: "0f53a0d3ef7e44ebb0365c800752d5bd" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st01.bj" port: 18700 } } I1031 23:05:22.609850 35074 raft_consensus.cc:319] T b4e182b626554f3597d5c733c80997e6 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1031 23:05:22.609863 35074 raft_consensus.cc:548] T b4e182b626554f3597d5c733c80997e6 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f53a0d3ef7e44ebb0365c800752d5bd, State: 1, Role: FOLLOWER I1031 23:05:22.609879 35074 consensus_queue.cc:176] T b4e182b626554f3597d5c733c80997e6 P 0f53a0d3ef7e44ebb0365c800752d5bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER I1031 23:05:22.631845 5857 ts_tablet_manager.cc:862] T 012b21c0f95e4dd594076018fd6fdb7e P 0f53a0d3ef7e44ebb0365c800752d5bd: Registered tablet (data state: TABLET_DATA_READY) I1031 23:05:22.632414 35079 ts_tablet_manager.cc:723] T 012b21c0f95e4dd594076018fd6fdb7e P 0f53a0d3ef7e44ebb0365c800752d5bd: Bootstrapping tablet I1031 23:05:22.632836 35079 tablet_bootstrap.cc:382] T 012b21c0f95e4dd594076018fd6fdb7e P 0f53a0d3ef7e44ebb0365c800752d5bd: Bootstrap starting. I1031 23:05:22.633016 35079 tablet_bootstrap.cc:541] T 012b21c0f95e4dd594076018fd6fdb7e P 0f53a0d3ef7e44ebb0365c800752d5bd: Time spent opening tablet: real 0.000s user 0.000s sys 0.000s I1031 23:05:22.633153 35079 tablet_bootstrap.cc:484] T 012b21c0f95e4dd594076018fd6fdb7e P 0f53a0d3ef7e44ebb0365c800752d5bd: No blocks or log segments found. Creating new log. I1031 23:05:22.634610 35079 tablet_bootstrap.cc:382] T 012b21c0f95e4dd594076018fd6fdb7e P 0f53a0d3ef7e44ebb0365c800752d5bd: No bootstrap required, opened a new log I1031 23:05:22.634642 35079 ts_tablet_manager.cc:728] T 012b21c0f95e4dd594076018fd6fdb7e P 0f53a0d3ef7e44ebb0365c800752d5bd: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s I1031 23:05:22.635105 35079 raft_consensus.cc:288] T 012b21c0f95e4dd594076018fd6fdb7e P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23f962e4a1744381ad5fa0d2d8b10241" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st07.bj" port: 18700 } } peers { permanent_uuid: "0f53a0d3ef7e44ebb0365c800752d5bd" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st01.bj" port: 18700 } } peers { permanent_uuid: "625f49b9554549e6a5c3a92b76e576cf" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st10.bj" port: 18700 } } I1031 23:05:22.635125 35079 raft_consensus.cc:319] T 012b21c0f95e4dd594076018fd6fdb7e P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1031 23:05:22.635133 35079 raft_consensus.cc:548] T 012b21c0f95e4dd594076018fd6fdb7e P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f53a0d3ef7e44ebb0365c800752d5bd, State: 1, Role: FOLLOWER I1031 23:05:22.635160 35079 consensus_queue.cc:176] T 012b21c0f95e4dd594076018fd6fdb7e P 0f53a0d3ef7e44ebb0365c800752d5bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER I1031 23:05:22.789542 5882 raft_consensus.cc:2162] T 95f15519e52a48edab3b765ae3a8e93d P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.732s I1031 23:05:22.789641 5882 raft_consensus.cc:1747] T 95f15519e52a48edab3b765ae3a8e93d P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3eb30bd1bc34b95902fa61724239553 in term 0. I1031 23:05:22.791340 5882 raft_consensus.cc:2206] T 95f15519e52a48edab3b765ae3a8e93d P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Advancing to term 1 I1031 23:05:22.791396 5882 raft_consensus.cc:2162] T 95f15519e52a48edab3b765ae3a8e93d P 0f53a0d3ef7e44ebb0365c800752d5bd [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.994s I1031 23:05:22.792062 5882 raft_consensus.cc:1747] T 95f15519e52a48edab3b765ae3a8e93d P 0f53a0d3ef7e44ebb0365c800752d5bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3eb30bd1bc34b95902fa61724239553 in term 1. W1031 23:05:22.846271 5816 consensus_peers.cc:357] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 0a8bbcbbb5a6459aa5258464755db3bb (c3-kudu-tst-st02.bj:18700): Couldn't send request to peer 0a8bbcbbb5a6459aa5258464755db3bb for tablet 648eb376255a4dd4bb5818262efcc813. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 180 times. I1031 23:05:23.003857 35077 raft_consensus.cc:411] T b4e182b626554f3597d5c733c80997e6 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I1031 23:05:23.003890 35077 raft_consensus.cc:2162] T b4e182b626554f3597d5c733c80997e6 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.569s I1031 23:05:23.003932 35077 raft_consensus.cc:435] T b4e182b626554f3597d5c733c80997e6 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3eb30bd1bc34b95902fa61724239553" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st05.bj" port: 18700 } } peers { permanent_uuid: "193181be29284287b25df2ee92938f0b" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st08.bj" port: 18700 } } peers { permanent_uuid: "0f53a0d3ef7e44ebb0365c800752d5bd" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st01.bj" port: 18700 } } I1031 23:05:23.004664 35077 leader_election.cc:216] T b4e182b626554f3597d5c733c80997e6 P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer d3eb30bd1bc34b95902fa61724239553 I1031 23:05:23.004703 35077 leader_election.cc:216] T b4e182b626554f3597d5c733c80997e6 P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 193181be29284287b25df2ee92938f0b W1031 23:05:23.004998 5815 leader_election.cc:278] T b4e182b626554f3597d5c733c80997e6 P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 193181be29284287b25df2ee92938f0b: Illegal state: Tablet not RUNNING: NOT_STARTED I1031 23:05:23.005198 5815 leader_election.cc:348] T b4e182b626554f3597d5c733c80997e6 P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Vote granted by peer d3eb30bd1bc34b95902fa61724239553 I1031 23:05:23.005218 5815 leader_election.cc:243] T b4e182b626554f3597d5c733c80997e6 P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. I1031 23:05:23.005303 35078 raft_consensus.cc:2162] T b4e182b626554f3597d5c733c80997e6 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.620s I1031 23:05:23.005327 35078 raft_consensus.cc:1971] T b4e182b626554f3597d5c733c80997e6 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Leader pre-election won for term 1 I1031 23:05:23.005349 35078 raft_consensus.cc:411] T b4e182b626554f3597d5c733c80997e6 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I1031 23:05:23.005360 35078 raft_consensus.cc:2162] T b4e182b626554f3597d5c733c80997e6 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.901s I1031 23:05:23.005368 35078 raft_consensus.cc:2206] T b4e182b626554f3597d5c733c80997e6 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Advancing to term 1 I1031 23:05:23.005816 35078 raft_consensus.cc:435] T b4e182b626554f3597d5c733c80997e6 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3eb30bd1bc34b95902fa61724239553" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st05.bj" port: 18700 } } peers { permanent_uuid: "193181be29284287b25df2ee92938f0b" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st08.bj" port: 18700 } } peers { permanent_uuid: "0f53a0d3ef7e44ebb0365c800752d5bd" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st01.bj" port: 18700 } } I1031 23:05:23.006335 35078 leader_election.cc:216] T b4e182b626554f3597d5c733c80997e6 P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 election: Requesting vote from peer d3eb30bd1bc34b95902fa61724239553 I1031 23:05:23.006368 35078 leader_election.cc:216] T b4e182b626554f3597d5c733c80997e6 P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 election: Requesting vote from peer 193181be29284287b25df2ee92938f0b W1031 23:05:23.006681 5815 leader_election.cc:278] T b4e182b626554f3597d5c733c80997e6 P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 193181be29284287b25df2ee92938f0b: Illegal state: Tablet not RUNNING: NOT_STARTED I1031 23:05:23.007576 5815 leader_election.cc:348] T b4e182b626554f3597d5c733c80997e6 P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 election: Vote granted by peer d3eb30bd1bc34b95902fa61724239553 I1031 23:05:23.007596 5815 leader_election.cc:243] T b4e182b626554f3597d5c733c80997e6 P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I1031 23:05:23.007632 35078 raft_consensus.cc:2162] T b4e182b626554f3597d5c733c80997e6 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.771s I1031 23:05:23.007648 35078 raft_consensus.cc:1971] T b4e182b626554f3597d5c733c80997e6 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 1 FOLLOWER]: Leader election won for term 1 I1031 23:05:23.008442 35078 raft_consensus.cc:518] T b4e182b626554f3597d5c733c80997e6 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 1 LEADER]: Becoming Leader. State: Replica: 0f53a0d3ef7e44ebb0365c800752d5bd, State: 1, Role: LEADER I1031 23:05:23.008533 35078 consensus_queue.cc:158] T b4e182b626554f3597d5c733c80997e6 P 0f53a0d3ef7e44ebb0365c800752d5bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Current term: 1, Majority size: 2, State: 1, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3eb30bd1bc34b95902fa61724239553" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st05.bj" port: 18700 } } peers { permanent_uuid: "193181be29284287b25df2ee92938f0b" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st08.bj" port: 18700 } } peers { permanent_uuid: "0f53a0d3ef7e44ebb0365c800752d5bd" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st01.bj" port: 18700 } } W1031 23:05:23.079248 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 120 times. W1031 23:05:23.095141 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 180 times. I1031 23:05:23.183188 35084 raft_consensus.cc:411] T 012b21c0f95e4dd594076018fd6fdb7e P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I1031 23:05:23.183254 35084 raft_consensus.cc:2162] T 012b21c0f95e4dd594076018fd6fdb7e P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.801s I1031 23:05:23.183317 35084 raft_consensus.cc:435] T 012b21c0f95e4dd594076018fd6fdb7e P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23f962e4a1744381ad5fa0d2d8b10241" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st07.bj" port: 18700 } } peers { permanent_uuid: "0f53a0d3ef7e44ebb0365c800752d5bd" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st01.bj" port: 18700 } } peers { permanent_uuid: "625f49b9554549e6a5c3a92b76e576cf" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st10.bj" port: 18700 } } I1031 23:05:23.183962 35084 leader_election.cc:216] T 012b21c0f95e4dd594076018fd6fdb7e P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 23f962e4a1744381ad5fa0d2d8b10241 I1031 23:05:23.183995 35084 leader_election.cc:216] T 012b21c0f95e4dd594076018fd6fdb7e P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 625f49b9554549e6a5c3a92b76e576cf W1031 23:05:23.184283 5816 leader_election.cc:278] T 012b21c0f95e4dd594076018fd6fdb7e P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 23f962e4a1744381ad5fa0d2d8b10241: Illegal state: Tablet not RUNNING: NOT_STARTED W1031 23:05:23.184355 5814 leader_election.cc:278] T 012b21c0f95e4dd594076018fd6fdb7e P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 625f49b9554549e6a5c3a92b76e576cf: Illegal state: Tablet not RUNNING: NOT_STARTED I1031 23:05:23.184381 5814 leader_election.cc:243] T 012b21c0f95e4dd594076018fd6fdb7e P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. I1031 23:05:23.184437 35085 raft_consensus.cc:2162] T 012b21c0f95e4dd594076018fd6fdb7e P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.572s I1031 23:05:23.184476 35085 raft_consensus.cc:1908] T 012b21c0f95e4dd594076018fd6fdb7e P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority I1031 23:05:23.293786 5886 raft_consensus.cc:918] T 95f15519e52a48edab3b765ae3a8e93d P 0f53a0d3ef7e44ebb0365c800752d5bd [term 1 FOLLOWER]: Refusing update from remote peer d3eb30bd1bc34b95902fa61724239553: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch) W1031 23:05:23.346426 5816 consensus_peers.cc:357] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 0a8bbcbbb5a6459aa5258464755db3bb (c3-kudu-tst-st02.bj:18700): Couldn't send request to peer 0a8bbcbbb5a6459aa5258464755db3bb for tablet 648eb376255a4dd4bb5818262efcc813. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 181 times. I1031 23:05:23.509783 35078 consensus_queue.cc:695] T b4e182b626554f3597d5c733c80997e6 P 0f53a0d3ef7e44ebb0365c800752d5bd [LEADER]: Connected to new peer: Peer: d3eb30bd1bc34b95902fa61724239553, Is new: false, Last received: 0.0, Next index: 1, Last known committed idx: 0, Last exchange result: ERROR, Needs tablet copy: false W1031 23:05:23.510280 5815 consensus_peers.cc:357] T b4e182b626554f3597d5c733c80997e6 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 193181be29284287b25df2ee92938f0b (c3-kudu-tst-st08.bj:18700): Couldn't send request to peer 193181be29284287b25df2ee92938f0b for tablet b4e182b626554f3597d5c733c80997e6. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 1 times. W1031 23:05:23.579457 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 121 times. W1031 23:05:23.595223 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 181 times. W1031 23:05:23.846469 5816 consensus_peers.cc:357] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 0a8bbcbbb5a6459aa5258464755db3bb (c3-kudu-tst-st02.bj:18700): Couldn't send request to peer 0a8bbcbbb5a6459aa5258464755db3bb for tablet 648eb376255a4dd4bb5818262efcc813. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 182 times. W1031 23:05:24.010464 5815 consensus_peers.cc:357] T b4e182b626554f3597d5c733c80997e6 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 193181be29284287b25df2ee92938f0b (c3-kudu-tst-st08.bj:18700): Couldn't send request to peer 193181be29284287b25df2ee92938f0b for tablet b4e182b626554f3597d5c733c80997e6. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 2 times. W1031 23:05:24.079525 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 122 times. W1031 23:05:24.095327 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 182 times. W1031 23:05:24.310559 5815 consensus_peers.cc:357] T b4e182b626554f3597d5c733c80997e6 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 193181be29284287b25df2ee92938f0b (c3-kudu-tst-st08.bj:18700): Couldn't send request to peer 193181be29284287b25df2ee92938f0b for tablet b4e182b626554f3597d5c733c80997e6. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 3 times. W1031 23:05:24.346478 5816 consensus_peers.cc:357] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 0a8bbcbbb5a6459aa5258464755db3bb (c3-kudu-tst-st02.bj:18700): Couldn't send request to peer 0a8bbcbbb5a6459aa5258464755db3bb for tablet 648eb376255a4dd4bb5818262efcc813. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 183 times. W1031 23:05:24.579612 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 123 times. W1031 23:05:24.595383 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 183 times. W1031 23:05:24.810590 5815 consensus_peers.cc:357] T b4e182b626554f3597d5c733c80997e6 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 193181be29284287b25df2ee92938f0b (c3-kudu-tst-st08.bj:18700): Couldn't send request to peer 193181be29284287b25df2ee92938f0b for tablet b4e182b626554f3597d5c733c80997e6. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 4 times. W1031 23:05:24.846655 5816 consensus_peers.cc:357] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 0a8bbcbbb5a6459aa5258464755db3bb (c3-kudu-tst-st02.bj:18700): Couldn't send request to peer 0a8bbcbbb5a6459aa5258464755db3bb for tablet 648eb376255a4dd4bb5818262efcc813. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 184 times. W1031 23:05:25.079627 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 124 times. W1031 23:05:25.095490 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 184 times. W1031 23:05:25.310647 5815 consensus_peers.cc:357] T b4e182b626554f3597d5c733c80997e6 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 193181be29284287b25df2ee92938f0b (c3-kudu-tst-st08.bj:18700): Couldn't send request to peer 193181be29284287b25df2ee92938f0b for tablet b4e182b626554f3597d5c733c80997e6. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 5 times. W1031 23:05:25.346699 5816 consensus_peers.cc:357] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 0a8bbcbbb5a6459aa5258464755db3bb (c3-kudu-tst-st02.bj:18700): Couldn't send request to peer 0a8bbcbbb5a6459aa5258464755db3bb for tablet 648eb376255a4dd4bb5818262efcc813. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 185 times. I1031 23:05:25.469280 5857 tablet_service.cc:691] Processing DeleteTablet for tablet 117cd74bcdab4942901acb0005cdd702 with delete_type TABLET_DATA_DELETED (Replaced by acb401e40e7049fbbfdbeb40474353d9 at 2017-10-31 23:04:52 CST) from {username='work'} at 10.108.187.10:53997 I1031 23:05:25.543488 5857 tablet_service.cc:691] Processing DeleteTablet for tablet 6a813ca355f749b6b31a773aa0f2ac7a with delete_type TABLET_DATA_DELETED (Replaced by 582c1281bf5147b2bd40c9a950a1152f at 2017-10-31 23:04:22 CST) from {username='work'} at 10.108.187.10:53997 I1031 23:05:25.546959 5857 tablet_service.cc:691] Processing DeleteTablet for tablet bff7dab34ccd4b42842077e82106f6fb with delete_type TABLET_DATA_DELETED (Replaced by 005a48fb466c4a52896842dd00124755 at 2017-10-31 23:04:52 CST) from {username='work'} at 10.108.187.10:53997 W1031 23:05:25.579701 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 125 times. I1031 23:05:25.586838 5857 tablet_service.cc:691] Processing DeleteTablet for tablet 56155a2adc24450d89e338cc955bfa18 with delete_type TABLET_DATA_DELETED (Replaced by 80ba7f33ec864c5f96c61932edb5cdc4 at 2017-10-31 23:04:52 CST) from {username='work'} at 10.108.187.10:53997 W1031 23:05:25.595484 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 185 times. W1031 23:05:25.810714 5815 consensus_peers.cc:357] T b4e182b626554f3597d5c733c80997e6 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 193181be29284287b25df2ee92938f0b (c3-kudu-tst-st08.bj:18700): Couldn't send request to peer 193181be29284287b25df2ee92938f0b for tablet b4e182b626554f3597d5c733c80997e6. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 6 times. W1031 23:05:25.846742 5816 consensus_peers.cc:357] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 0a8bbcbbb5a6459aa5258464755db3bb (c3-kudu-tst-st02.bj:18700): Couldn't send request to peer 0a8bbcbbb5a6459aa5258464755db3bb for tablet 648eb376255a4dd4bb5818262efcc813. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 186 times. W1031 23:05:26.079810 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 126 times. W1031 23:05:26.095548 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 186 times. W1031 23:05:26.310813 5815 consensus_peers.cc:357] T b4e182b626554f3597d5c733c80997e6 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 193181be29284287b25df2ee92938f0b (c3-kudu-tst-st08.bj:18700): Couldn't send request to peer 193181be29284287b25df2ee92938f0b for tablet b4e182b626554f3597d5c733c80997e6. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 7 times. W1031 23:05:26.346861 5816 consensus_peers.cc:357] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 0a8bbcbbb5a6459aa5258464755db3bb (c3-kudu-tst-st02.bj:18700): Couldn't send request to peer 0a8bbcbbb5a6459aa5258464755db3bb for tablet 648eb376255a4dd4bb5818262efcc813. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 187 times. I1031 23:05:26.398499 35084 raft_consensus.cc:411] T 012b21c0f95e4dd594076018fd6fdb7e P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I1031 23:05:26.398550 35084 raft_consensus.cc:2162] T 012b21c0f95e4dd594076018fd6fdb7e P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 2.739s I1031 23:05:26.398624 35084 raft_consensus.cc:435] T 012b21c0f95e4dd594076018fd6fdb7e P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23f962e4a1744381ad5fa0d2d8b10241" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st07.bj" port: 18700 } } peers { permanent_uuid: "0f53a0d3ef7e44ebb0365c800752d5bd" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st01.bj" port: 18700 } } peers { permanent_uuid: "625f49b9554549e6a5c3a92b76e576cf" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st10.bj" port: 18700 } } I1031 23:05:26.399309 35084 leader_election.cc:216] T 012b21c0f95e4dd594076018fd6fdb7e P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 23f962e4a1744381ad5fa0d2d8b10241 I1031 23:05:26.399350 35084 leader_election.cc:216] T 012b21c0f95e4dd594076018fd6fdb7e P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 625f49b9554549e6a5c3a92b76e576cf W1031 23:05:26.399639 5814 leader_election.cc:278] T 012b21c0f95e4dd594076018fd6fdb7e P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 625f49b9554549e6a5c3a92b76e576cf: Illegal state: Tablet not RUNNING: NOT_STARTED W1031 23:05:26.399690 5816 leader_election.cc:278] T 012b21c0f95e4dd594076018fd6fdb7e P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 23f962e4a1744381ad5fa0d2d8b10241: Illegal state: Tablet not RUNNING: NOT_STARTED I1031 23:05:26.399718 5816 leader_election.cc:243] T 012b21c0f95e4dd594076018fd6fdb7e P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. I1031 23:05:26.400002 35107 raft_consensus.cc:2162] T 012b21c0f95e4dd594076018fd6fdb7e P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 3.218s I1031 23:05:26.400033 35107 raft_consensus.cc:1908] T 012b21c0f95e4dd594076018fd6fdb7e P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority W1031 23:05:26.579968 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 127 times. W1031 23:05:26.595710 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 187 times. W1031 23:05:26.810914 5815 consensus_peers.cc:357] T b4e182b626554f3597d5c733c80997e6 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 193181be29284287b25df2ee92938f0b (c3-kudu-tst-st08.bj:18700): Couldn't send request to peer 193181be29284287b25df2ee92938f0b for tablet b4e182b626554f3597d5c733c80997e6. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 8 times. W1031 23:05:26.846943 5816 consensus_peers.cc:357] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 0a8bbcbbb5a6459aa5258464755db3bb (c3-kudu-tst-st02.bj:18700): Couldn't send request to peer 0a8bbcbbb5a6459aa5258464755db3bb for tablet 648eb376255a4dd4bb5818262efcc813. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 188 times. W1031 23:05:27.079902 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 128 times. W1031 23:05:27.095721 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 188 times. W1031 23:05:27.311015 5815 consensus_peers.cc:357] T b4e182b626554f3597d5c733c80997e6 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 193181be29284287b25df2ee92938f0b (c3-kudu-tst-st08.bj:18700): Couldn't send request to peer 193181be29284287b25df2ee92938f0b for tablet b4e182b626554f3597d5c733c80997e6. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 9 times. W1031 23:05:27.346966 5816 consensus_peers.cc:357] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 0a8bbcbbb5a6459aa5258464755db3bb (c3-kudu-tst-st02.bj:18700): Couldn't send request to peer 0a8bbcbbb5a6459aa5258464755db3bb for tablet 648eb376255a4dd4bb5818262efcc813. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 189 times. W1031 23:05:27.580094 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 129 times. W1031 23:05:27.595839 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 189 times. W1031 23:05:27.811108 5815 consensus_peers.cc:357] T b4e182b626554f3597d5c733c80997e6 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 193181be29284287b25df2ee92938f0b (c3-kudu-tst-st08.bj:18700): Couldn't send request to peer 193181be29284287b25df2ee92938f0b for tablet b4e182b626554f3597d5c733c80997e6. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 10 times. W1031 23:05:27.846983 5816 consensus_peers.cc:357] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 0a8bbcbbb5a6459aa5258464755db3bb (c3-kudu-tst-st02.bj:18700): Couldn't send request to peer 0a8bbcbbb5a6459aa5258464755db3bb for tablet 648eb376255a4dd4bb5818262efcc813. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 190 times. W1031 23:05:28.080099 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 130 times. W1031 23:05:28.095939 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 190 times. W1031 23:05:28.311168 5815 consensus_peers.cc:357] T b4e182b626554f3597d5c733c80997e6 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 193181be29284287b25df2ee92938f0b (c3-kudu-tst-st08.bj:18700): Couldn't send request to peer 193181be29284287b25df2ee92938f0b for tablet b4e182b626554f3597d5c733c80997e6. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 11 times. W1031 23:05:28.347014 5816 consensus_peers.cc:357] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 0a8bbcbbb5a6459aa5258464755db3bb (c3-kudu-tst-st02.bj:18700): Couldn't send request to peer 0a8bbcbbb5a6459aa5258464755db3bb for tablet 648eb376255a4dd4bb5818262efcc813. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 191 times. W1031 23:05:28.580230 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 131 times. W1031 23:05:28.596040 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 191 times. W1031 23:05:28.811233 5815 consensus_peers.cc:357] T b4e182b626554f3597d5c733c80997e6 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 193181be29284287b25df2ee92938f0b (c3-kudu-tst-st08.bj:18700): Couldn't send request to peer 193181be29284287b25df2ee92938f0b for tablet b4e182b626554f3597d5c733c80997e6. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 12 times. W1031 23:05:28.847128 5816 consensus_peers.cc:357] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 0a8bbcbbb5a6459aa5258464755db3bb (c3-kudu-tst-st02.bj:18700): Couldn't send request to peer 0a8bbcbbb5a6459aa5258464755db3bb for tablet 648eb376255a4dd4bb5818262efcc813. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 192 times. W1031 23:05:29.080241 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 132 times. W1031 23:05:29.095990 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 192 times. W1031 23:05:29.311373 5815 consensus_peers.cc:357] T b4e182b626554f3597d5c733c80997e6 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 193181be29284287b25df2ee92938f0b (c3-kudu-tst-st08.bj:18700): Couldn't send request to peer 193181be29284287b25df2ee92938f0b for tablet b4e182b626554f3597d5c733c80997e6. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 13 times. W1031 23:05:29.347299 5816 consensus_peers.cc:357] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 0a8bbcbbb5a6459aa5258464755db3bb (c3-kudu-tst-st02.bj:18700): Couldn't send request to peer 0a8bbcbbb5a6459aa5258464755db3bb for tablet 648eb376255a4dd4bb5818262efcc813. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 193 times. W1031 23:05:29.580322 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 133 times. W1031 23:05:29.596135 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 193 times. W1031 23:05:29.811411 5815 consensus_peers.cc:357] T b4e182b626554f3597d5c733c80997e6 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 193181be29284287b25df2ee92938f0b (c3-kudu-tst-st08.bj:18700): Couldn't send request to peer 193181be29284287b25df2ee92938f0b for tablet b4e182b626554f3597d5c733c80997e6. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 14 times. W1031 23:05:29.847262 5816 consensus_peers.cc:357] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 0a8bbcbbb5a6459aa5258464755db3bb (c3-kudu-tst-st02.bj:18700): Couldn't send request to peer 0a8bbcbbb5a6459aa5258464755db3bb for tablet 648eb376255a4dd4bb5818262efcc813. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 194 times. W1031 23:05:30.080308 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 134 times. W1031 23:05:30.096298 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 194 times. W1031 23:05:30.311442 5815 consensus_peers.cc:357] T b4e182b626554f3597d5c733c80997e6 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 193181be29284287b25df2ee92938f0b (c3-kudu-tst-st08.bj:18700): Couldn't send request to peer 193181be29284287b25df2ee92938f0b for tablet b4e182b626554f3597d5c733c80997e6. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 15 times. W1031 23:05:30.347380 5816 consensus_peers.cc:357] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 0a8bbcbbb5a6459aa5258464755db3bb (c3-kudu-tst-st02.bj:18700): Couldn't send request to peer 0a8bbcbbb5a6459aa5258464755db3bb for tablet 648eb376255a4dd4bb5818262efcc813. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 195 times. W1031 23:05:30.580466 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 135 times. W1031 23:05:30.596206 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 195 times. W1031 23:05:30.811445 5815 consensus_peers.cc:357] T b4e182b626554f3597d5c733c80997e6 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 193181be29284287b25df2ee92938f0b (c3-kudu-tst-st08.bj:18700): Couldn't send request to peer 193181be29284287b25df2ee92938f0b for tablet b4e182b626554f3597d5c733c80997e6. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 16 times. W1031 23:05:30.847453 5816 consensus_peers.cc:357] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 0a8bbcbbb5a6459aa5258464755db3bb (c3-kudu-tst-st02.bj:18700): Couldn't send request to peer 0a8bbcbbb5a6459aa5258464755db3bb for tablet 648eb376255a4dd4bb5818262efcc813. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 196 times. W1031 23:05:31.080513 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 136 times. W1031 23:05:31.096340 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 196 times. I1031 23:05:31.156996 35084 raft_consensus.cc:411] T 012b21c0f95e4dd594076018fd6fdb7e P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I1031 23:05:31.157032 35084 raft_consensus.cc:2162] T 012b21c0f95e4dd594076018fd6fdb7e P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 3.582s I1031 23:05:31.157081 35084 raft_consensus.cc:435] T 012b21c0f95e4dd594076018fd6fdb7e P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23f962e4a1744381ad5fa0d2d8b10241" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st07.bj" port: 18700 } } peers { permanent_uuid: "0f53a0d3ef7e44ebb0365c800752d5bd" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st01.bj" port: 18700 } } peers { permanent_uuid: "625f49b9554549e6a5c3a92b76e576cf" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st10.bj" port: 18700 } } I1031 23:05:31.157601 35084 leader_election.cc:216] T 012b21c0f95e4dd594076018fd6fdb7e P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 23f962e4a1744381ad5fa0d2d8b10241 I1031 23:05:31.157627 35084 leader_election.cc:216] T 012b21c0f95e4dd594076018fd6fdb7e P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 625f49b9554549e6a5c3a92b76e576cf W1031 23:05:31.157867 5816 leader_election.cc:278] T 012b21c0f95e4dd594076018fd6fdb7e P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 23f962e4a1744381ad5fa0d2d8b10241: Illegal state: Tablet not RUNNING: NOT_STARTED W1031 23:05:31.157974 5814 leader_election.cc:278] T 012b21c0f95e4dd594076018fd6fdb7e P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 625f49b9554549e6a5c3a92b76e576cf: Illegal state: Tablet not RUNNING: NOT_STARTED I1031 23:05:31.158006 5814 leader_election.cc:243] T 012b21c0f95e4dd594076018fd6fdb7e P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. I1031 23:05:31.158272 35113 raft_consensus.cc:2162] T 012b21c0f95e4dd594076018fd6fdb7e P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.958s I1031 23:05:31.158288 35113 raft_consensus.cc:1908] T 012b21c0f95e4dd594076018fd6fdb7e P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority W1031 23:05:31.311604 5815 consensus_peers.cc:357] T b4e182b626554f3597d5c733c80997e6 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 193181be29284287b25df2ee92938f0b (c3-kudu-tst-st08.bj:18700): Couldn't send request to peer 193181be29284287b25df2ee92938f0b for tablet b4e182b626554f3597d5c733c80997e6. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 17 times. W1031 23:05:31.347676 5816 consensus_peers.cc:357] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 0a8bbcbbb5a6459aa5258464755db3bb (c3-kudu-tst-st02.bj:18700): Couldn't send request to peer 0a8bbcbbb5a6459aa5258464755db3bb for tablet 648eb376255a4dd4bb5818262efcc813. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 197 times. W1031 23:05:31.580605 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 137 times. I1031 23:05:31.593586 34585 raft_consensus.cc:411] T 56155a2adc24450d89e338cc955bfa18 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I1031 23:05:31.593624 34585 raft_consensus.cc:2162] T 56155a2adc24450d89e338cc955bfa18 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 4.693s I1031 23:05:31.593683 34585 raft_consensus.cc:435] T 56155a2adc24450d89e338cc955bfa18 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625f49b9554549e6a5c3a92b76e576cf" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st10.bj" port: 18700 } } peers { permanent_uuid: "193181be29284287b25df2ee92938f0b" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st08.bj" port: 18700 } } peers { permanent_uuid: "0f53a0d3ef7e44ebb0365c800752d5bd" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st01.bj" port: 18700 } } I1031 23:05:31.594241 34585 leader_election.cc:216] T 56155a2adc24450d89e338cc955bfa18 P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 625f49b9554549e6a5c3a92b76e576cf I1031 23:05:31.594285 34585 leader_election.cc:216] T 56155a2adc24450d89e338cc955bfa18 P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 193181be29284287b25df2ee92938f0b W1031 23:05:31.594564 5815 leader_election.cc:278] T 56155a2adc24450d89e338cc955bfa18 P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 193181be29284287b25df2ee92938f0b: Illegal state: Tablet not RUNNING: NOT_STARTED W1031 23:05:31.594606 5814 leader_election.cc:278] T 56155a2adc24450d89e338cc955bfa18 P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 625f49b9554549e6a5c3a92b76e576cf: Illegal state: Tablet not RUNNING: NOT_STARTED I1031 23:05:31.594635 5814 leader_election.cc:243] T 56155a2adc24450d89e338cc955bfa18 P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. I1031 23:05:31.594892 35114 raft_consensus.cc:2162] T 56155a2adc24450d89e338cc955bfa18 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 14.012s I1031 23:05:31.594915 35114 raft_consensus.cc:1908] T 56155a2adc24450d89e338cc955bfa18 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority W1031 23:05:31.596421 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 197 times. W1031 23:05:31.811693 5815 consensus_peers.cc:357] T b4e182b626554f3597d5c733c80997e6 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 193181be29284287b25df2ee92938f0b (c3-kudu-tst-st08.bj:18700): Couldn't send request to peer 193181be29284287b25df2ee92938f0b for tablet b4e182b626554f3597d5c733c80997e6. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 18 times. W1031 23:05:31.847596 5816 consensus_peers.cc:357] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 0a8bbcbbb5a6459aa5258464755db3bb (c3-kudu-tst-st02.bj:18700): Couldn't send request to peer 0a8bbcbbb5a6459aa5258464755db3bb for tablet 648eb376255a4dd4bb5818262efcc813. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 198 times. W1031 23:05:32.080621 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 138 times. W1031 23:05:32.096593 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 198 times. W1031 23:05:32.311729 5815 consensus_peers.cc:357] T b4e182b626554f3597d5c733c80997e6 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 193181be29284287b25df2ee92938f0b (c3-kudu-tst-st08.bj:18700): Couldn't send request to peer 193181be29284287b25df2ee92938f0b for tablet b4e182b626554f3597d5c733c80997e6. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 19 times. W1031 23:05:32.347697 5816 consensus_peers.cc:357] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 0a8bbcbbb5a6459aa5258464755db3bb (c3-kudu-tst-st02.bj:18700): Couldn't send request to peer 0a8bbcbbb5a6459aa5258464755db3bb for tablet 648eb376255a4dd4bb5818262efcc813. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 199 times. W1031 23:05:32.580786 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 139 times. W1031 23:05:32.596693 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 199 times. W1031 23:05:32.811882 5815 consensus_peers.cc:357] T b4e182b626554f3597d5c733c80997e6 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 193181be29284287b25df2ee92938f0b (c3-kudu-tst-st08.bj:18700): Couldn't send request to peer 193181be29284287b25df2ee92938f0b for tablet b4e182b626554f3597d5c733c80997e6. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 20 times. W1031 23:05:32.847651 5816 consensus_peers.cc:357] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 0a8bbcbbb5a6459aa5258464755db3bb (c3-kudu-tst-st02.bj:18700): Couldn't send request to peer 0a8bbcbbb5a6459aa5258464755db3bb for tablet 648eb376255a4dd4bb5818262efcc813. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 200 times. W1031 23:05:33.080844 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 140 times. W1031 23:05:33.096810 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 200 times. W1031 23:05:33.311925 5815 consensus_peers.cc:357] T b4e182b626554f3597d5c733c80997e6 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 193181be29284287b25df2ee92938f0b (c3-kudu-tst-st08.bj:18700): Couldn't send request to peer 193181be29284287b25df2ee92938f0b for tablet b4e182b626554f3597d5c733c80997e6. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 21 times. W1031 23:05:33.347893 5816 consensus_peers.cc:357] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 0a8bbcbbb5a6459aa5258464755db3bb (c3-kudu-tst-st02.bj:18700): Couldn't send request to peer 0a8bbcbbb5a6459aa5258464755db3bb for tablet 648eb376255a4dd4bb5818262efcc813. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 201 times. W1031 23:05:33.580971 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 141 times. W1031 23:05:33.596909 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 201 times. W1031 23:05:33.811971 5815 consensus_peers.cc:357] T b4e182b626554f3597d5c733c80997e6 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 193181be29284287b25df2ee92938f0b (c3-kudu-tst-st08.bj:18700): Couldn't send request to peer 193181be29284287b25df2ee92938f0b for tablet b4e182b626554f3597d5c733c80997e6. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 22 times. W1031 23:05:33.847827 5816 consensus_peers.cc:357] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 0a8bbcbbb5a6459aa5258464755db3bb (c3-kudu-tst-st02.bj:18700): Couldn't send request to peer 0a8bbcbbb5a6459aa5258464755db3bb for tablet 648eb376255a4dd4bb5818262efcc813. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 202 times. W1031 23:05:34.080946 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 142 times. I1031 23:05:34.083956 34590 raft_consensus.cc:411] T 117cd74bcdab4942901acb0005cdd702 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I1031 23:05:34.083992 34590 raft_consensus.cc:2162] T 117cd74bcdab4942901acb0005cdd702 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 5.624s I1031 23:05:34.084058 34590 raft_consensus.cc:435] T 117cd74bcdab4942901acb0005cdd702 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625f49b9554549e6a5c3a92b76e576cf" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st10.bj" port: 18700 } } peers { permanent_uuid: "0a8bbcbbb5a6459aa5258464755db3bb" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st02.bj" port: 18700 } } peers { permanent_uuid: "0f53a0d3ef7e44ebb0365c800752d5bd" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st01.bj" port: 18700 } } I1031 23:05:34.084681 34590 leader_election.cc:216] T 117cd74bcdab4942901acb0005cdd702 P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 625f49b9554549e6a5c3a92b76e576cf I1031 23:05:34.084704 34590 leader_election.cc:216] T 117cd74bcdab4942901acb0005cdd702 P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 0a8bbcbbb5a6459aa5258464755db3bb W1031 23:05:34.084952 5816 leader_election.cc:278] T 117cd74bcdab4942901acb0005cdd702 P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0a8bbcbbb5a6459aa5258464755db3bb: Not found: Tablet not found W1031 23:05:34.085002 5814 leader_election.cc:278] T 117cd74bcdab4942901acb0005cdd702 P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 625f49b9554549e6a5c3a92b76e576cf: Illegal state: Tablet not RUNNING: NOT_STARTED I1031 23:05:34.085031 5814 leader_election.cc:243] T 117cd74bcdab4942901acb0005cdd702 P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. I1031 23:05:34.085289 35120 raft_consensus.cc:2162] T 117cd74bcdab4942901acb0005cdd702 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 2.469s I1031 23:05:34.085306 35120 raft_consensus.cc:1908] T 117cd74bcdab4942901acb0005cdd702 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority W1031 23:05:34.096866 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 202 times. I1031 23:05:34.121623 5885 raft_consensus.cc:1708] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate 0a8bbcbbb5a6459aa5258464755db3bb for term 1 because replica is either leader or believes a valid leader to be alive. W1031 23:05:34.312042 5815 consensus_peers.cc:357] T b4e182b626554f3597d5c733c80997e6 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 193181be29284287b25df2ee92938f0b (c3-kudu-tst-st08.bj:18700): Couldn't send request to peer 193181be29284287b25df2ee92938f0b for tablet b4e182b626554f3597d5c733c80997e6. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 23 times. I1031 23:05:34.347975 34142 consensus_queue.cc:695] T 648eb376255a4dd4bb5818262efcc813 P 0f53a0d3ef7e44ebb0365c800752d5bd [LEADER]: Connected to new peer: Peer: 0a8bbcbbb5a6459aa5258464755db3bb, Is new: false, Last received: 0.0, Next index: 1, Last known committed idx: 0, Last exchange result: ERROR, Needs tablet copy: false W1031 23:05:34.580982 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 143 times. I1031 23:05:34.584288 34067 raft_consensus.cc:411] T 6a813ca355f749b6b31a773aa0f2ac7a P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I1031 23:05:34.584339 34067 raft_consensus.cc:2162] T 6a813ca355f749b6b31a773aa0f2ac7a P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 8.861s I1031 23:05:34.584390 34067 raft_consensus.cc:435] T 6a813ca355f749b6b31a773aa0f2ac7a P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a8bbcbbb5a6459aa5258464755db3bb" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st02.bj" port: 18700 } } peers { permanent_uuid: "0f53a0d3ef7e44ebb0365c800752d5bd" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st01.bj" port: 18700 } } peers { permanent_uuid: "193181be29284287b25df2ee92938f0b" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st08.bj" port: 18700 } } I1031 23:05:34.585041 34067 leader_election.cc:216] T 6a813ca355f749b6b31a773aa0f2ac7a P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 0a8bbcbbb5a6459aa5258464755db3bb I1031 23:05:34.585078 34067 leader_election.cc:216] T 6a813ca355f749b6b31a773aa0f2ac7a P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 193181be29284287b25df2ee92938f0b W1031 23:05:34.585242 5816 leader_election.cc:278] T 6a813ca355f749b6b31a773aa0f2ac7a P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0a8bbcbbb5a6459aa5258464755db3bb: Not found: Tablet not found W1031 23:05:34.585399 5815 leader_election.cc:278] T 6a813ca355f749b6b31a773aa0f2ac7a P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 193181be29284287b25df2ee92938f0b: Illegal state: Tablet not RUNNING: NOT_STARTED I1031 23:05:34.585422 5815 leader_election.cc:243] T 6a813ca355f749b6b31a773aa0f2ac7a P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. I1031 23:05:34.585893 35145 raft_consensus.cc:2162] T 6a813ca355f749b6b31a773aa0f2ac7a P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 10.018s I1031 23:05:34.585912 35145 raft_consensus.cc:1908] T 6a813ca355f749b6b31a773aa0f2ac7a P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority W1031 23:05:34.596913 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 203 times. I1031 23:05:34.711164 34574 raft_consensus.cc:411] T bff7dab34ccd4b42842077e82106f6fb P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I1031 23:05:34.711220 34574 raft_consensus.cc:2162] T bff7dab34ccd4b42842077e82106f6fb P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 11.219s I1031 23:05:34.711282 34574 raft_consensus.cc:435] T bff7dab34ccd4b42842077e82106f6fb P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f53a0d3ef7e44ebb0365c800752d5bd" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st01.bj" port: 18700 } } peers { permanent_uuid: "23f962e4a1744381ad5fa0d2d8b10241" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st07.bj" port: 18700 } } peers { permanent_uuid: "193181be29284287b25df2ee92938f0b" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st08.bj" port: 18700 } } I1031 23:05:34.711975 34574 leader_election.cc:216] T bff7dab34ccd4b42842077e82106f6fb P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 23f962e4a1744381ad5fa0d2d8b10241 I1031 23:05:34.712018 34574 leader_election.cc:216] T bff7dab34ccd4b42842077e82106f6fb P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 193181be29284287b25df2ee92938f0b W1031 23:05:34.712296 5815 leader_election.cc:278] T bff7dab34ccd4b42842077e82106f6fb P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 193181be29284287b25df2ee92938f0b: Illegal state: Tablet not RUNNING: NOT_STARTED W1031 23:05:34.712352 5816 leader_election.cc:278] T bff7dab34ccd4b42842077e82106f6fb P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 23f962e4a1744381ad5fa0d2d8b10241: Illegal state: Tablet not RUNNING: NOT_STARTED I1031 23:05:34.712376 5816 leader_election.cc:243] T bff7dab34ccd4b42842077e82106f6fb P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. I1031 23:05:34.713245 35161 raft_consensus.cc:2162] T bff7dab34ccd4b42842077e82106f6fb P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 15.344s I1031 23:05:34.713275 35161 raft_consensus.cc:1908] T bff7dab34ccd4b42842077e82106f6fb P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority W1031 23:05:34.812180 5815 consensus_peers.cc:357] T b4e182b626554f3597d5c733c80997e6 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 193181be29284287b25df2ee92938f0b (c3-kudu-tst-st08.bj:18700): Couldn't send request to peer 193181be29284287b25df2ee92938f0b for tablet b4e182b626554f3597d5c733c80997e6. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 24 times. W1031 23:05:35.081092 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 144 times. W1031 23:05:35.097000 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 204 times. I1031 23:05:35.131322 35084 raft_consensus.cc:411] T 012b21c0f95e4dd594076018fd6fdb7e P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I1031 23:05:35.131363 35084 raft_consensus.cc:2162] T 012b21c0f95e4dd594076018fd6fdb7e P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 4.996s I1031 23:05:35.131420 35084 raft_consensus.cc:435] T 012b21c0f95e4dd594076018fd6fdb7e P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23f962e4a1744381ad5fa0d2d8b10241" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st07.bj" port: 18700 } } peers { permanent_uuid: "0f53a0d3ef7e44ebb0365c800752d5bd" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st01.bj" port: 18700 } } peers { permanent_uuid: "625f49b9554549e6a5c3a92b76e576cf" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st10.bj" port: 18700 } } I1031 23:05:35.131988 35084 leader_election.cc:216] T 012b21c0f95e4dd594076018fd6fdb7e P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 23f962e4a1744381ad5fa0d2d8b10241 I1031 23:05:35.132016 35084 leader_election.cc:216] T 012b21c0f95e4dd594076018fd6fdb7e P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 625f49b9554549e6a5c3a92b76e576cf W1031 23:05:35.132297 5814 leader_election.cc:278] T 012b21c0f95e4dd594076018fd6fdb7e P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 625f49b9554549e6a5c3a92b76e576cf: Illegal state: Tablet not RUNNING: NOT_STARTED W1031 23:05:35.132346 5816 leader_election.cc:278] T 012b21c0f95e4dd594076018fd6fdb7e P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 23f962e4a1744381ad5fa0d2d8b10241: Illegal state: Tablet not RUNNING: NOT_STARTED I1031 23:05:35.132372 5816 leader_election.cc:243] T 012b21c0f95e4dd594076018fd6fdb7e P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. I1031 23:05:35.132633 35162 raft_consensus.cc:2162] T 012b21c0f95e4dd594076018fd6fdb7e P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 4.934s I1031 23:05:35.132649 35162 raft_consensus.cc:1908] T 012b21c0f95e4dd594076018fd6fdb7e P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority W1031 23:05:35.312216 5815 consensus_peers.cc:357] T b4e182b626554f3597d5c733c80997e6 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 193181be29284287b25df2ee92938f0b (c3-kudu-tst-st08.bj:18700): Couldn't send request to peer 193181be29284287b25df2ee92938f0b for tablet b4e182b626554f3597d5c733c80997e6. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 25 times. W1031 23:05:35.581162 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 145 times. W1031 23:05:35.597096 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 205 times. W1031 23:05:35.812280 5815 consensus_peers.cc:357] T b4e182b626554f3597d5c733c80997e6 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 193181be29284287b25df2ee92938f0b (c3-kudu-tst-st08.bj:18700): Couldn't send request to peer 193181be29284287b25df2ee92938f0b for tablet b4e182b626554f3597d5c733c80997e6. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 26 times. W1031 23:05:36.081239 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 146 times. W1031 23:05:36.097132 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 206 times. W1031 23:05:36.312335 5815 consensus_peers.cc:357] T b4e182b626554f3597d5c733c80997e6 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 193181be29284287b25df2ee92938f0b (c3-kudu-tst-st08.bj:18700): Couldn't send request to peer 193181be29284287b25df2ee92938f0b for tablet b4e182b626554f3597d5c733c80997e6. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 27 times. W1031 23:05:36.581337 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 147 times. W1031 23:05:36.597285 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 207 times. W1031 23:05:36.812360 5815 consensus_peers.cc:357] T b4e182b626554f3597d5c733c80997e6 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 193181be29284287b25df2ee92938f0b (c3-kudu-tst-st08.bj:18700): Couldn't send request to peer 193181be29284287b25df2ee92938f0b for tablet b4e182b626554f3597d5c733c80997e6. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 28 times. W1031 23:05:37.081482 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 148 times. W1031 23:05:37.097296 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 208 times. W1031 23:05:37.312497 5815 consensus_peers.cc:357] T b4e182b626554f3597d5c733c80997e6 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 193181be29284287b25df2ee92938f0b (c3-kudu-tst-st08.bj:18700): Couldn't send request to peer 193181be29284287b25df2ee92938f0b for tablet b4e182b626554f3597d5c733c80997e6. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 29 times. W1031 23:05:37.581595 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 149 times. W1031 23:05:37.597414 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 209 times. W1031 23:05:37.812521 5815 consensus_peers.cc:357] T b4e182b626554f3597d5c733c80997e6 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 193181be29284287b25df2ee92938f0b (c3-kudu-tst-st08.bj:18700): Couldn't send request to peer 193181be29284287b25df2ee92938f0b for tablet b4e182b626554f3597d5c733c80997e6. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 30 times. W1031 23:05:38.081583 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 150 times. W1031 23:05:38.097486 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 210 times. W1031 23:05:38.312629 5815 consensus_peers.cc:357] T b4e182b626554f3597d5c733c80997e6 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 193181be29284287b25df2ee92938f0b (c3-kudu-tst-st08.bj:18700): Couldn't send request to peer 193181be29284287b25df2ee92938f0b for tablet b4e182b626554f3597d5c733c80997e6. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 31 times. W1031 23:05:38.581748 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 151 times. W1031 23:05:38.597528 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 211 times. I1031 23:05:38.721312 34590 raft_consensus.cc:411] T 117cd74bcdab4942901acb0005cdd702 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I1031 23:05:38.721356 34590 raft_consensus.cc:2162] T 117cd74bcdab4942901acb0005cdd702 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 12.794s I1031 23:05:38.721416 34590 raft_consensus.cc:435] T 117cd74bcdab4942901acb0005cdd702 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625f49b9554549e6a5c3a92b76e576cf" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st10.bj" port: 18700 } } peers { permanent_uuid: "0a8bbcbbb5a6459aa5258464755db3bb" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st02.bj" port: 18700 } } peers { permanent_uuid: "0f53a0d3ef7e44ebb0365c800752d5bd" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st01.bj" port: 18700 } } I1031 23:05:38.721951 34590 leader_election.cc:216] T 117cd74bcdab4942901acb0005cdd702 P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 625f49b9554549e6a5c3a92b76e576cf I1031 23:05:38.721982 34590 leader_election.cc:216] T 117cd74bcdab4942901acb0005cdd702 P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 0a8bbcbbb5a6459aa5258464755db3bb W1031 23:05:38.722220 5816 leader_election.cc:278] T 117cd74bcdab4942901acb0005cdd702 P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0a8bbcbbb5a6459aa5258464755db3bb: Not found: Tablet not found W1031 23:05:38.722342 5814 leader_election.cc:278] T 117cd74bcdab4942901acb0005cdd702 P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 625f49b9554549e6a5c3a92b76e576cf: Illegal state: Tablet not RUNNING: NOT_STARTED I1031 23:05:38.722363 5814 leader_election.cc:243] T 117cd74bcdab4942901acb0005cdd702 P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. I1031 23:05:38.722693 35170 raft_consensus.cc:2162] T 117cd74bcdab4942901acb0005cdd702 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 2.735s I1031 23:05:38.722714 35170 raft_consensus.cc:1908] T 117cd74bcdab4942901acb0005cdd702 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority W1031 23:05:38.812639 5815 consensus_peers.cc:357] T b4e182b626554f3597d5c733c80997e6 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 193181be29284287b25df2ee92938f0b (c3-kudu-tst-st08.bj:18700): Couldn't send request to peer 193181be29284287b25df2ee92938f0b for tablet b4e182b626554f3597d5c733c80997e6. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 32 times. W1031 23:05:39.081779 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 152 times. W1031 23:05:39.097697 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 212 times. W1031 23:05:39.312638 5815 consensus_peers.cc:357] T b4e182b626554f3597d5c733c80997e6 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 193181be29284287b25df2ee92938f0b (c3-kudu-tst-st08.bj:18700): Couldn't send request to peer 193181be29284287b25df2ee92938f0b for tablet b4e182b626554f3597d5c733c80997e6. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 33 times. W1031 23:05:39.581902 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 153 times. W1031 23:05:39.597743 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 213 times. W1031 23:05:39.812827 5815 consensus_peers.cc:357] T b4e182b626554f3597d5c733c80997e6 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 193181be29284287b25df2ee92938f0b (c3-kudu-tst-st08.bj:18700): Couldn't send request to peer 193181be29284287b25df2ee92938f0b for tablet b4e182b626554f3597d5c733c80997e6. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 34 times. W1031 23:05:40.082016 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 154 times. W1031 23:05:40.097695 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 214 times. W1031 23:05:40.312903 5815 consensus_peers.cc:357] T b4e182b626554f3597d5c733c80997e6 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 193181be29284287b25df2ee92938f0b (c3-kudu-tst-st08.bj:18700): Couldn't send request to peer 193181be29284287b25df2ee92938f0b for tablet b4e182b626554f3597d5c733c80997e6. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 35 times. W1031 23:05:40.582073 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 155 times. W1031 23:05:40.597936 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 215 times. W1031 23:05:40.813035 5815 consensus_peers.cc:357] T b4e182b626554f3597d5c733c80997e6 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 193181be29284287b25df2ee92938f0b (c3-kudu-tst-st08.bj:18700): Couldn't send request to peer 193181be29284287b25df2ee92938f0b for tablet b4e182b626554f3597d5c733c80997e6. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 36 times. W1031 23:05:41.082092 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 156 times. W1031 23:05:41.097915 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 216 times. W1031 23:05:41.313020 5815 consensus_peers.cc:357] T b4e182b626554f3597d5c733c80997e6 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 193181be29284287b25df2ee92938f0b (c3-kudu-tst-st08.bj:18700): Couldn't send request to peer 193181be29284287b25df2ee92938f0b for tablet b4e182b626554f3597d5c733c80997e6. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 37 times. W1031 23:05:41.582248 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 157 times. W1031 23:05:41.597981 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 217 times. W1031 23:05:41.813128 5815 consensus_peers.cc:357] T b4e182b626554f3597d5c733c80997e6 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 193181be29284287b25df2ee92938f0b (c3-kudu-tst-st08.bj:18700): Couldn't send request to peer 193181be29284287b25df2ee92938f0b for tablet b4e182b626554f3597d5c733c80997e6. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 38 times. I1031 23:05:41.964049 35084 raft_consensus.cc:411] T 012b21c0f95e4dd594076018fd6fdb7e P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I1031 23:05:41.964089 35084 raft_consensus.cc:2162] T 012b21c0f95e4dd594076018fd6fdb7e P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 3.477s I1031 23:05:41.964143 35084 raft_consensus.cc:435] T 012b21c0f95e4dd594076018fd6fdb7e P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23f962e4a1744381ad5fa0d2d8b10241" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st07.bj" port: 18700 } } peers { permanent_uuid: "0f53a0d3ef7e44ebb0365c800752d5bd" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st01.bj" port: 18700 } } peers { permanent_uuid: "625f49b9554549e6a5c3a92b76e576cf" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st10.bj" port: 18700 } } I1031 23:05:41.964664 35084 leader_election.cc:216] T 012b21c0f95e4dd594076018fd6fdb7e P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 23f962e4a1744381ad5fa0d2d8b10241 I1031 23:05:41.964692 35084 leader_election.cc:216] T 012b21c0f95e4dd594076018fd6fdb7e P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 625f49b9554549e6a5c3a92b76e576cf W1031 23:05:41.964901 5816 leader_election.cc:278] T 012b21c0f95e4dd594076018fd6fdb7e P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 23f962e4a1744381ad5fa0d2d8b10241: Illegal state: Tablet not RUNNING: NOT_STARTED W1031 23:05:41.965003 5814 leader_election.cc:278] T 012b21c0f95e4dd594076018fd6fdb7e P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 625f49b9554549e6a5c3a92b76e576cf: Illegal state: Tablet not RUNNING: NOT_STARTED I1031 23:05:41.965026 5814 leader_election.cc:243] T 012b21c0f95e4dd594076018fd6fdb7e P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. I1031 23:05:41.965855 35178 raft_consensus.cc:2162] T 012b21c0f95e4dd594076018fd6fdb7e P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 4.808s I1031 23:05:41.965876 35178 raft_consensus.cc:1908] T 012b21c0f95e4dd594076018fd6fdb7e P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority W1031 23:05:42.082262 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 158 times. W1031 23:05:42.098002 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 218 times. W1031 23:05:42.313133 5815 consensus_peers.cc:357] T b4e182b626554f3597d5c733c80997e6 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 193181be29284287b25df2ee92938f0b (c3-kudu-tst-st08.bj:18700): Couldn't send request to peer 193181be29284287b25df2ee92938f0b for tablet b4e182b626554f3597d5c733c80997e6. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 39 times. W1031 23:05:42.582413 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 159 times. W1031 23:05:42.598088 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 219 times. W1031 23:05:42.813210 5815 consensus_peers.cc:357] T b4e182b626554f3597d5c733c80997e6 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 193181be29284287b25df2ee92938f0b (c3-kudu-tst-st08.bj:18700): Couldn't send request to peer 193181be29284287b25df2ee92938f0b for tablet b4e182b626554f3597d5c733c80997e6. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 40 times. W1031 23:05:43.082377 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 160 times. W1031 23:05:43.098222 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 220 times. I1031 23:05:43.237738 34590 raft_consensus.cc:411] T 117cd74bcdab4942901acb0005cdd702 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I1031 23:05:43.237792 34590 raft_consensus.cc:2162] T 117cd74bcdab4942901acb0005cdd702 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 12.933s I1031 23:05:43.237851 34590 raft_consensus.cc:435] T 117cd74bcdab4942901acb0005cdd702 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625f49b9554549e6a5c3a92b76e576cf" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st10.bj" port: 18700 } } peers { permanent_uuid: "0a8bbcbbb5a6459aa5258464755db3bb" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st02.bj" port: 18700 } } peers { permanent_uuid: "0f53a0d3ef7e44ebb0365c800752d5bd" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st01.bj" port: 18700 } } I1031 23:05:43.238462 34590 leader_election.cc:216] T 117cd74bcdab4942901acb0005cdd702 P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 625f49b9554549e6a5c3a92b76e576cf I1031 23:05:43.238489 34590 leader_election.cc:216] T 117cd74bcdab4942901acb0005cdd702 P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 0a8bbcbbb5a6459aa5258464755db3bb W1031 23:05:43.238757 5814 leader_election.cc:278] T 117cd74bcdab4942901acb0005cdd702 P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 625f49b9554549e6a5c3a92b76e576cf: Illegal state: Tablet not RUNNING: NOT_STARTED W1031 23:05:43.238790 5816 leader_election.cc:278] T 117cd74bcdab4942901acb0005cdd702 P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0a8bbcbbb5a6459aa5258464755db3bb: Not found: Tablet not found I1031 23:05:43.238816 5816 leader_election.cc:243] T 117cd74bcdab4942901acb0005cdd702 P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. I1031 23:05:43.239069 35181 raft_consensus.cc:2162] T 117cd74bcdab4942901acb0005cdd702 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 13.115s I1031 23:05:43.239091 35181 raft_consensus.cc:1908] T 117cd74bcdab4942901acb0005cdd702 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority W1031 23:05:43.313311 5815 consensus_peers.cc:357] T b4e182b626554f3597d5c733c80997e6 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 193181be29284287b25df2ee92938f0b (c3-kudu-tst-st08.bj:18700): Couldn't send request to peer 193181be29284287b25df2ee92938f0b for tablet b4e182b626554f3597d5c733c80997e6. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 41 times. W1031 23:05:43.582485 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 161 times. W1031 23:05:43.598263 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 221 times. W1031 23:05:43.813406 5815 consensus_peers.cc:357] T b4e182b626554f3597d5c733c80997e6 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 193181be29284287b25df2ee92938f0b (c3-kudu-tst-st08.bj:18700): Couldn't send request to peer 193181be29284287b25df2ee92938f0b for tablet b4e182b626554f3597d5c733c80997e6. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 42 times. W1031 23:05:44.082533 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 162 times. W1031 23:05:44.098392 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 222 times. W1031 23:05:44.313477 5815 consensus_peers.cc:357] T b4e182b626554f3597d5c733c80997e6 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 193181be29284287b25df2ee92938f0b (c3-kudu-tst-st08.bj:18700): Couldn't send request to peer 193181be29284287b25df2ee92938f0b for tablet b4e182b626554f3597d5c733c80997e6. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 43 times. W1031 23:05:44.582715 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 163 times. W1031 23:05:44.598475 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 223 times. W1031 23:05:44.813524 5815 consensus_peers.cc:357] T b4e182b626554f3597d5c733c80997e6 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 193181be29284287b25df2ee92938f0b (c3-kudu-tst-st08.bj:18700): Couldn't send request to peer 193181be29284287b25df2ee92938f0b for tablet b4e182b626554f3597d5c733c80997e6. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 44 times. W1031 23:05:45.082712 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 164 times. W1031 23:05:45.098477 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 224 times. W1031 23:05:45.313638 5815 consensus_peers.cc:357] T b4e182b626554f3597d5c733c80997e6 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 193181be29284287b25df2ee92938f0b (c3-kudu-tst-st08.bj:18700): Couldn't send request to peer 193181be29284287b25df2ee92938f0b for tablet b4e182b626554f3597d5c733c80997e6. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 45 times. W1031 23:05:45.582808 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 165 times. W1031 23:05:45.598623 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 225 times. W1031 23:05:45.813745 5815 consensus_peers.cc:357] T b4e182b626554f3597d5c733c80997e6 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 193181be29284287b25df2ee92938f0b (c3-kudu-tst-st08.bj:18700): Couldn't send request to peer 193181be29284287b25df2ee92938f0b for tablet b4e182b626554f3597d5c733c80997e6. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 46 times. W1031 23:05:46.082820 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 166 times. W1031 23:05:46.098703 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 226 times. I1031 23:05:46.308656 34067 raft_consensus.cc:411] T 6a813ca355f749b6b31a773aa0f2ac7a P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I1031 23:05:46.308697 34067 raft_consensus.cc:2162] T 6a813ca355f749b6b31a773aa0f2ac7a P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 8.403s I1031 23:05:46.308750 34067 raft_consensus.cc:435] T 6a813ca355f749b6b31a773aa0f2ac7a P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a8bbcbbb5a6459aa5258464755db3bb" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st02.bj" port: 18700 } } peers { permanent_uuid: "0f53a0d3ef7e44ebb0365c800752d5bd" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st01.bj" port: 18700 } } peers { permanent_uuid: "193181be29284287b25df2ee92938f0b" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st08.bj" port: 18700 } } I1031 23:05:46.309429 34067 leader_election.cc:216] T 6a813ca355f749b6b31a773aa0f2ac7a P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 0a8bbcbbb5a6459aa5258464755db3bb I1031 23:05:46.309458 34067 leader_election.cc:216] T 6a813ca355f749b6b31a773aa0f2ac7a P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 193181be29284287b25df2ee92938f0b W1031 23:05:46.309660 5816 leader_election.cc:278] T 6a813ca355f749b6b31a773aa0f2ac7a P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0a8bbcbbb5a6459aa5258464755db3bb: Not found: Tablet not found W1031 23:05:46.309790 5815 leader_election.cc:278] T 6a813ca355f749b6b31a773aa0f2ac7a P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 193181be29284287b25df2ee92938f0b: Illegal state: Tablet not RUNNING: NOT_STARTED I1031 23:05:46.309810 5815 leader_election.cc:243] T 6a813ca355f749b6b31a773aa0f2ac7a P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. I1031 23:05:46.310086 35199 raft_consensus.cc:2162] T 6a813ca355f749b6b31a773aa0f2ac7a P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 7.861s I1031 23:05:46.310102 35199 raft_consensus.cc:1908] T 6a813ca355f749b6b31a773aa0f2ac7a P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority W1031 23:05:46.313730 5815 consensus_peers.cc:357] T b4e182b626554f3597d5c733c80997e6 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 193181be29284287b25df2ee92938f0b (c3-kudu-tst-st08.bj:18700): Couldn't send request to peer 193181be29284287b25df2ee92938f0b for tablet b4e182b626554f3597d5c733c80997e6. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 47 times. W1031 23:05:46.582831 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 167 times. W1031 23:05:46.598724 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 227 times. W1031 23:05:46.813827 5815 consensus_peers.cc:357] T b4e182b626554f3597d5c733c80997e6 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 193181be29284287b25df2ee92938f0b (c3-kudu-tst-st08.bj:18700): Couldn't send request to peer 193181be29284287b25df2ee92938f0b for tablet b4e182b626554f3597d5c733c80997e6. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 48 times. W1031 23:05:47.082936 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 168 times. W1031 23:05:47.098861 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 228 times. I1031 23:05:47.306716 34585 raft_consensus.cc:411] T 56155a2adc24450d89e338cc955bfa18 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I1031 23:05:47.306753 34585 raft_consensus.cc:2162] T 56155a2adc24450d89e338cc955bfa18 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 6.989s I1031 23:05:47.306813 34585 raft_consensus.cc:435] T 56155a2adc24450d89e338cc955bfa18 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625f49b9554549e6a5c3a92b76e576cf" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st10.bj" port: 18700 } } peers { permanent_uuid: "193181be29284287b25df2ee92938f0b" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st08.bj" port: 18700 } } peers { permanent_uuid: "0f53a0d3ef7e44ebb0365c800752d5bd" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st01.bj" port: 18700 } } I1031 23:05:47.307381 34585 leader_election.cc:216] T 56155a2adc24450d89e338cc955bfa18 P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 625f49b9554549e6a5c3a92b76e576cf I1031 23:05:47.307409 34585 leader_election.cc:216] T 56155a2adc24450d89e338cc955bfa18 P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 193181be29284287b25df2ee92938f0b W1031 23:05:47.307731 5814 leader_election.cc:278] T 56155a2adc24450d89e338cc955bfa18 P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 625f49b9554549e6a5c3a92b76e576cf: Illegal state: Tablet not RUNNING: NOT_STARTED W1031 23:05:47.307785 5815 leader_election.cc:278] T 56155a2adc24450d89e338cc955bfa18 P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 193181be29284287b25df2ee92938f0b: Illegal state: Tablet not RUNNING: NOT_STARTED I1031 23:05:47.307802 5815 leader_election.cc:243] T 56155a2adc24450d89e338cc955bfa18 P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. I1031 23:05:47.308596 35200 raft_consensus.cc:2162] T 56155a2adc24450d89e338cc955bfa18 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 14.845s I1031 23:05:47.308617 35200 raft_consensus.cc:1908] T 56155a2adc24450d89e338cc955bfa18 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority W1031 23:05:47.313895 5815 consensus_peers.cc:357] T b4e182b626554f3597d5c733c80997e6 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 193181be29284287b25df2ee92938f0b (c3-kudu-tst-st08.bj:18700): Couldn't send request to peer 193181be29284287b25df2ee92938f0b for tablet b4e182b626554f3597d5c733c80997e6. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 49 times. W1031 23:05:47.583021 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 169 times. W1031 23:05:47.598927 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 229 times. W1031 23:05:47.814028 5815 consensus_peers.cc:357] T b4e182b626554f3597d5c733c80997e6 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 193181be29284287b25df2ee92938f0b (c3-kudu-tst-st08.bj:18700): Couldn't send request to peer 193181be29284287b25df2ee92938f0b for tablet b4e182b626554f3597d5c733c80997e6. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 50 times. W1031 23:05:48.083139 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 170 times. W1031 23:05:48.099052 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 230 times. W1031 23:05:48.314066 5815 consensus_peers.cc:357] T b4e182b626554f3597d5c733c80997e6 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 193181be29284287b25df2ee92938f0b (c3-kudu-tst-st08.bj:18700): Couldn't send request to peer 193181be29284287b25df2ee92938f0b for tablet b4e182b626554f3597d5c733c80997e6. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 51 times. W1031 23:05:48.583106 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 171 times. W1031 23:05:48.599037 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 231 times. I1031 23:05:48.717746 35084 raft_consensus.cc:411] T 012b21c0f95e4dd594076018fd6fdb7e P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I1031 23:05:48.717784 35084 raft_consensus.cc:2162] T 012b21c0f95e4dd594076018fd6fdb7e P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 15.992s I1031 23:05:48.717835 35084 raft_consensus.cc:435] T 012b21c0f95e4dd594076018fd6fdb7e P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23f962e4a1744381ad5fa0d2d8b10241" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st07.bj" port: 18700 } } peers { permanent_uuid: "0f53a0d3ef7e44ebb0365c800752d5bd" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st01.bj" port: 18700 } } peers { permanent_uuid: "625f49b9554549e6a5c3a92b76e576cf" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st10.bj" port: 18700 } } I1031 23:05:48.718382 35084 leader_election.cc:216] T 012b21c0f95e4dd594076018fd6fdb7e P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 23f962e4a1744381ad5fa0d2d8b10241 I1031 23:05:48.718410 35084 leader_election.cc:216] T 012b21c0f95e4dd594076018fd6fdb7e P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 625f49b9554549e6a5c3a92b76e576cf W1031 23:05:48.718688 5816 leader_election.cc:278] T 012b21c0f95e4dd594076018fd6fdb7e P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 23f962e4a1744381ad5fa0d2d8b10241: Illegal state: Tablet not RUNNING: NOT_STARTED W1031 23:05:48.718791 5814 leader_election.cc:278] T 012b21c0f95e4dd594076018fd6fdb7e P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 625f49b9554549e6a5c3a92b76e576cf: Illegal state: Tablet not RUNNING: NOT_STARTED I1031 23:05:48.718817 5814 leader_election.cc:243] T 012b21c0f95e4dd594076018fd6fdb7e P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. I1031 23:05:48.719079 35203 raft_consensus.cc:2162] T 012b21c0f95e4dd594076018fd6fdb7e P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 12.067s I1031 23:05:48.719097 35203 raft_consensus.cc:1908] T 012b21c0f95e4dd594076018fd6fdb7e P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority W1031 23:05:48.814235 5815 consensus_peers.cc:357] T b4e182b626554f3597d5c733c80997e6 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 193181be29284287b25df2ee92938f0b (c3-kudu-tst-st08.bj:18700): Couldn't send request to peer 193181be29284287b25df2ee92938f0b for tablet b4e182b626554f3597d5c733c80997e6. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 52 times. W1031 23:05:49.083169 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 172 times. W1031 23:05:49.099150 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 232 times. W1031 23:05:49.314198 5815 consensus_peers.cc:357] T b4e182b626554f3597d5c733c80997e6 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 193181be29284287b25df2ee92938f0b (c3-kudu-tst-st08.bj:18700): Couldn't send request to peer 193181be29284287b25df2ee92938f0b for tablet b4e182b626554f3597d5c733c80997e6. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 53 times. W1031 23:05:49.583323 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 173 times. W1031 23:05:49.599340 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 233 times. W1031 23:05:49.814352 5815 consensus_peers.cc:357] T b4e182b626554f3597d5c733c80997e6 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 193181be29284287b25df2ee92938f0b (c3-kudu-tst-st08.bj:18700): Couldn't send request to peer 193181be29284287b25df2ee92938f0b for tablet b4e182b626554f3597d5c733c80997e6. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 54 times. W1031 23:05:50.083360 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 174 times. W1031 23:05:50.099309 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 234 times. W1031 23:05:50.314393 5815 consensus_peers.cc:357] T b4e182b626554f3597d5c733c80997e6 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 193181be29284287b25df2ee92938f0b (c3-kudu-tst-st08.bj:18700): Couldn't send request to peer 193181be29284287b25df2ee92938f0b for tablet b4e182b626554f3597d5c733c80997e6. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 55 times. W1031 23:05:50.583477 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 175 times. W1031 23:05:50.599346 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 235 times. W1031 23:05:50.814399 5815 consensus_peers.cc:357] T b4e182b626554f3597d5c733c80997e6 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 193181be29284287b25df2ee92938f0b (c3-kudu-tst-st08.bj:18700): Couldn't send request to peer 193181be29284287b25df2ee92938f0b for tablet b4e182b626554f3597d5c733c80997e6. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 56 times. W1031 23:05:51.083525 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 176 times. W1031 23:05:51.099462 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 236 times. W1031 23:05:51.314600 5815 consensus_peers.cc:357] T b4e182b626554f3597d5c733c80997e6 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 193181be29284287b25df2ee92938f0b (c3-kudu-tst-st08.bj:18700): Couldn't send request to peer 193181be29284287b25df2ee92938f0b for tablet b4e182b626554f3597d5c733c80997e6. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 57 times. I1031 23:05:51.417315 5853 maintenance_manager.cc:208] P 0f53a0d3ef7e44ebb0365c800752d5bd: Unregistered op CompactRowSetsOp(6a813ca355f749b6b31a773aa0f2ac7a) I1031 23:05:51.417379 5853 maintenance_manager.cc:208] P 0f53a0d3ef7e44ebb0365c800752d5bd: Unregistered op MinorDeltaCompactionOp(6a813ca355f749b6b31a773aa0f2ac7a) I1031 23:05:51.417407 5853 maintenance_manager.cc:208] P 0f53a0d3ef7e44ebb0365c800752d5bd: Unregistered op MajorDeltaCompactionOp(6a813ca355f749b6b31a773aa0f2ac7a) I1031 23:05:51.417629 35069 ts_tablet_manager.cc:755] T 95f15519e52a48edab3b765ae3a8e93d P 0f53a0d3ef7e44ebb0365c800752d5bd: Time spent starting tablet: real 28.833s user 0.000s sys 0.002s W1031 23:05:51.417654 35069 ts_tablet_manager.cc:785] T 95f15519e52a48edab3b765ae3a8e93d P 0f53a0d3ef7e44ebb0365c800752d5bd: Tablet startup took 28833ms I1031 23:05:51.417693 5853 maintenance_manager.cc:208] P 0f53a0d3ef7e44ebb0365c800752d5bd: Unregistered op UndoDeltaBlockGCOp(6a813ca355f749b6b31a773aa0f2ac7a) W1031 23:05:51.417747 35069 ts_tablet_manager.cc:787] T 95f15519e52a48edab3b765ae3a8e93d P 0f53a0d3ef7e44ebb0365c800752d5bd: Trace: 1031 23:05:22.526192 (+ 0us) service_pool.cc:143] Inserting onto call queue 1031 23:05:22.526206 (+ 14us) service_pool.cc:202] Handling call 1031 23:05:22.526410 (+ 204us) ts_tablet_manager.cc:275] Acquired tablet manager lock 1031 23:05:22.526413 (+ 3us) ts_tablet_manager.cc:288] Creating new metadata... 1031 23:05:22.580936 (+ 54523us) tablet_metadata.cc:461] Metadata flushed 1031 23:05:22.582195 (+ 1259us) inbound_call.cc:130] Queueing success response 1031 23:05:22.582242 (+ 47us) ts_tablet_manager.cc:724] Bootstrapping tablet 1031 23:05:22.584572 (+ 2330us) ts_tablet_manager.cc:756] Initializing tablet peer 1031 23:05:22.584589 (+ 17us) tablet_peer.cc:161] Creating consensus instance 1031 23:05:22.584900 (+ 311us) tablet_peer.cc:183] Starting instrumentation 1031 23:05:22.584904 (+ 4us) tablet_peer.cc:188] TabletPeer::Init() finished 1031 23:05:22.584905 (+ 1us) ts_tablet_manager.cc:771] Starting tablet peer 1031 23:05:22.584906 (+ 1us) tablet_peer.cc:195] Starting consensus Metrics: {"fdatasync":1,"fdatasync_us":36884,"mutex_wait_us":28831738,"raft.queue_time_us":391,"raft.run_cpu_time_us":3,"raft.run_wall_time_us":4,"tablet-open.queue_time_us":811,"tcmalloc_contention_cycles":3584,"thread_start_us":1550,"threads_started":3} I1031 23:05:51.417771 34582 ts_tablet_manager.cc:755] T 56155a2adc24450d89e338cc955bfa18 P 0f53a0d3ef7e44ebb0365c800752d5bd: Time spent starting tablet: real 88.820s user 0.001s sys 0.000s W1031 23:05:51.417788 34582 ts_tablet_manager.cc:785] T 56155a2adc24450d89e338cc955bfa18 P 0f53a0d3ef7e44ebb0365c800752d5bd: Tablet startup took 88819ms I1031 23:05:51.417819 34587 ts_tablet_manager.cc:755] T 117cd74bcdab4942901acb0005cdd702 P 0f53a0d3ef7e44ebb0365c800752d5bd: Time spent starting tablet: real 88.795s user 0.000s sys 0.001s W1031 23:05:51.417845 34587 ts_tablet_manager.cc:785] T 117cd74bcdab4942901acb0005cdd702 P 0f53a0d3ef7e44ebb0365c800752d5bd: Tablet startup took 88795ms W1031 23:05:51.417877 34582 ts_tablet_manager.cc:787] T 56155a2adc24450d89e338cc955bfa18 P 0f53a0d3ef7e44ebb0365c800752d5bd: Trace: 1031 23:04:22.477111 (+ 0us) service_pool.cc:143] Inserting onto call queue 1031 23:04:22.477135 (+ 24us) service_pool.cc:202] Handling call 1031 23:04:22.477381 (+ 246us) ts_tablet_manager.cc:275] Acquired tablet manager lock 1031 23:04:22.477387 (+ 6us) ts_tablet_manager.cc:288] Creating new metadata... 1031 23:04:22.594456 (+117069us) tablet_metadata.cc:461] Metadata flushed 1031 23:04:22.595687 (+ 1231us) inbound_call.cc:130] Queueing success response 1031 23:04:22.595724 (+ 37us) ts_tablet_manager.cc:724] Bootstrapping tablet 1031 23:04:22.597991 (+ 2267us) ts_tablet_manager.cc:756] Initializing tablet peer 1031 23:04:22.598008 (+ 17us) tablet_peer.cc:161] Creating consensus instance 1031 23:04:22.598319 (+ 311us) tablet_peer.cc:183] Starting instrumentation 1031 23:04:22.598323 (+ 4us) tablet_peer.cc:188] TabletPeer::Init() finished 1031 23:04:22.598323 (+ 0us) ts_tablet_manager.cc:771] Starting tablet peer 1031 23:04:22.598324 (+ 1us) tablet_peer.cc:195] Starting consensus Metrics: {"fdatasync":1,"fdatasync_us":40570,"mutex_wait_us":88818373,"raft.queue_time_us":421,"raft.run_cpu_time_us":4,"raft.run_wall_time_us":4,"tablet-open.queue_time_us":586,"tcmalloc_contention_cycles":239616,"thread_start_us":1396,"threads_started":3} I1031 23:05:51.417927 34576 ts_tablet_manager.cc:755] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd: Time spent starting tablet: real 88.846s user 0.000s sys 0.001s W1031 23:05:51.417958 34576 ts_tablet_manager.cc:785] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd: Tablet startup took 88846ms I1031 23:05:51.417997 34571 ts_tablet_manager.cc:755] T bff7dab34ccd4b42842077e82106f6fb P 0f53a0d3ef7e44ebb0365c800752d5bd: Time spent starting tablet: real 88.871s user 0.000s sys 0.002s W1031 23:05:51.418022 34571 ts_tablet_manager.cc:785] T bff7dab34ccd4b42842077e82106f6fb P 0f53a0d3ef7e44ebb0365c800752d5bd: Tablet startup took 88871ms W1031 23:05:51.418098 34576 ts_tablet_manager.cc:787] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd: Trace: 1031 23:04:22.477548 (+ 0us) service_pool.cc:143] Inserting onto call queue 1031 23:04:22.477564 (+ 16us) service_pool.cc:202] Handling call 1031 23:04:22.477731 (+ 167us) ts_tablet_manager.cc:275] Acquired tablet manager lock 1031 23:04:22.477739 (+ 8us) ts_tablet_manager.cc:288] Creating new metadata... 1031 23:04:22.569192 (+ 91453us) tablet_metadata.cc:461] Metadata flushed 1031 23:04:22.569885 (+ 693us) inbound_call.cc:130] Queueing success response 1031 23:04:22.569919 (+ 34us) ts_tablet_manager.cc:724] Bootstrapping tablet 1031 23:04:22.571623 (+ 1704us) ts_tablet_manager.cc:756] Initializing tablet peer 1031 23:04:22.571639 (+ 16us) tablet_peer.cc:161] Creating consensus instance 1031 23:04:22.571873 (+ 234us) tablet_peer.cc:183] Starting instrumentation 1031 23:04:22.571876 (+ 3us) tablet_peer.cc:188] TabletPeer::Init() finished 1031 23:04:22.571877 (+ 1us) ts_tablet_manager.cc:771] Starting tablet peer 1031 23:04:22.571877 (+ 0us) tablet_peer.cc:195] Starting consensus Metrics: {"fdatasync":1,"fdatasync_us":40444,"mutex_wait_us":88845080,"raft.queue_time_us":399,"raft.run_cpu_time_us":3,"raft.run_wall_time_us":3,"tablet-open.queue_time_us":210,"tcmalloc_contention_cycles":39680,"thread_start_us":994,"threads_started":3} W1031 23:05:51.418118 34571 ts_tablet_manager.cc:787] T bff7dab34ccd4b42842077e82106f6fb P 0f53a0d3ef7e44ebb0365c800752d5bd: Trace: 1031 23:04:22.477087 (+ 0us) service_pool.cc:143] Inserting onto call queue 1031 23:04:22.477103 (+ 16us) service_pool.cc:202] Handling call 1031 23:04:22.477231 (+ 128us) ts_tablet_manager.cc:275] Acquired tablet manager lock 1031 23:04:22.477234 (+ 3us) ts_tablet_manager.cc:288] Creating new metadata... 1031 23:04:22.543979 (+ 66745us) tablet_metadata.cc:461] Metadata flushed 1031 23:04:22.544717 (+ 738us) inbound_call.cc:130] Queueing success response 1031 23:04:22.544772 (+ 55us) ts_tablet_manager.cc:724] Bootstrapping tablet 1031 23:04:22.546788 (+ 2016us) ts_tablet_manager.cc:756] Initializing tablet peer 1031 23:04:22.546805 (+ 17us) tablet_peer.cc:161] Creating consensus instance 1031 23:04:22.547076 (+ 271us) tablet_peer.cc:183] Starting instrumentation 1031 23:04:22.547080 (+ 4us) tablet_peer.cc:188] TabletPeer::Init() finished 1031 23:04:22.547080 (+ 0us) ts_tablet_manager.cc:771] Starting tablet peer 1031 23:04:22.547081 (+ 1us) tablet_peer.cc:195] Starting consensus Metrics: {"fdatasync":1,"fdatasync_us":40645,"mutex_wait_us":88869857,"raft.queue_time_us":391,"raft.run_cpu_time_us":12,"raft.run_wall_time_us":13,"tablet-open.queue_time_us":289,"tcmalloc_contention_cycles":20608,"thread_start_us":1078,"threads_started":3} I1031 23:05:51.418100 5853 maintenance_manager.cc:208] P 0f53a0d3ef7e44ebb0365c800752d5bd: Unregistered op FlushMRSOp(6a813ca355f749b6b31a773aa0f2ac7a) W1031 23:05:51.417928 34587 ts_tablet_manager.cc:787] T 117cd74bcdab4942901acb0005cdd702 P 0f53a0d3ef7e44ebb0365c800752d5bd: Trace: 1031 23:04:22.477138 (+ 0us) service_pool.cc:143] Inserting onto call queue 1031 23:04:22.477152 (+ 14us) service_pool.cc:202] Handling call 1031 23:04:22.477346 (+ 194us) ts_tablet_manager.cc:275] Acquired tablet manager lock 1031 23:04:22.477354 (+ 8us) ts_tablet_manager.cc:288] Creating new metadata... 1031 23:04:22.619638 (+142284us) tablet_metadata.cc:461] Metadata flushed 1031 23:04:22.620525 (+ 887us) inbound_call.cc:130] Queueing success response 1031 23:04:22.620555 (+ 30us) ts_tablet_manager.cc:724] Bootstrapping tablet 1031 23:04:22.622415 (+ 1860us) ts_tablet_manager.cc:756] Initializing tablet peer 1031 23:04:22.622431 (+ 16us) tablet_peer.cc:161] Creating consensus instance 1031 23:04:22.622699 (+ 268us) tablet_peer.cc:183] Starting instrumentation 1031 23:04:22.622702 (+ 3us) tablet_peer.cc:188] TabletPeer::Init() finished 1031 23:04:22.622703 (+ 1us) ts_tablet_manager.cc:771] Starting tablet peer 1031 23:04:22.622703 (+ 0us) tablet_peer.cc:195] Starting consensus Metrics: {"fdatasync":1,"fdatasync_us":40663,"mutex_wait_us":88794413,"raft.queue_time_us":407,"raft.run_cpu_time_us":11,"raft.run_wall_time_us":11,"tablet-open.queue_time_us":440,"tcmalloc_contention_cycles":41728,"thread_start_us":924,"threads_started":3} I1031 23:05:51.418170 5853 maintenance_manager.cc:208] P 0f53a0d3ef7e44ebb0365c800752d5bd: Unregistered op FlushDeltaMemStoresOp(6a813ca355f749b6b31a773aa0f2ac7a) I1031 23:05:51.418228 35079 ts_tablet_manager.cc:755] T 012b21c0f95e4dd594076018fd6fdb7e P 0f53a0d3ef7e44ebb0365c800752d5bd: Time spent starting tablet: real 28.784s user 0.001s sys 0.000s W1031 23:05:51.418257 35079 ts_tablet_manager.cc:785] T 012b21c0f95e4dd594076018fd6fdb7e P 0f53a0d3ef7e44ebb0365c800752d5bd: Tablet startup took 28783ms I1031 23:05:51.418324 35074 ts_tablet_manager.cc:755] T b4e182b626554f3597d5c733c80997e6 P 0f53a0d3ef7e44ebb0365c800752d5bd: Time spent starting tablet: real 28.809s user 0.001s sys 0.000s I1031 23:05:51.418330 5853 maintenance_manager.cc:208] P 0f53a0d3ef7e44ebb0365c800752d5bd: Unregistered op LogGCOp(6a813ca355f749b6b31a773aa0f2ac7a) W1031 23:05:51.418349 35074 ts_tablet_manager.cc:785] T b4e182b626554f3597d5c733c80997e6 P 0f53a0d3ef7e44ebb0365c800752d5bd: Tablet startup took 28809ms W1031 23:05:51.418341 35079 ts_tablet_manager.cc:787] T 012b21c0f95e4dd594076018fd6fdb7e P 0f53a0d3ef7e44ebb0365c800752d5bd: Trace: 1031 23:05:22.526164 (+ 0us) service_pool.cc:143] Inserting onto call queue 1031 23:05:22.526193 (+ 29us) service_pool.cc:202] Handling call 1031 23:05:22.526349 (+ 156us) ts_tablet_manager.cc:275] Acquired tablet manager lock 1031 23:05:22.526352 (+ 3us) ts_tablet_manager.cc:288] Creating new metadata... 1031 23:05:22.631328 (+104976us) tablet_metadata.cc:461] Metadata flushed 1031 23:05:22.632408 (+ 1080us) inbound_call.cc:130] Queueing success response 1031 23:05:22.632448 (+ 40us) ts_tablet_manager.cc:724] Bootstrapping tablet 1031 23:05:22.634651 (+ 2203us) ts_tablet_manager.cc:756] Initializing tablet peer 1031 23:05:22.634667 (+ 16us) tablet_peer.cc:161] Creating consensus instance 1031 23:05:22.634932 (+ 265us) tablet_peer.cc:183] Starting instrumentation 1031 23:05:22.634935 (+ 3us) tablet_peer.cc:188] TabletPeer::Init() finished 1031 23:05:22.634935 (+ 0us) ts_tablet_manager.cc:771] Starting tablet peer 1031 23:05:22.634936 (+ 1us) tablet_peer.cc:195] Starting consensus Metrics: {"fdatasync":1,"fdatasync_us":36920,"mutex_wait_us":28782407,"raft.queue_time_us":594,"raft.run_cpu_time_us":7,"raft.run_wall_time_us":8,"spinlock_wait_cycles":1408,"tablet-open.queue_time_us":509,"tcmalloc_contention_cycles":56320,"thread_start_us":1146,"threads_started":3} I1031 23:05:51.418370 5853 raft_consensus.cc:1569] T 6a813ca355f749b6b31a773aa0f2ac7a P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Raft consensus shutting down. I1031 23:05:51.418397 5853 raft_consensus.cc:1585] T 6a813ca355f749b6b31a773aa0f2ac7a P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Raft consensus is shut down! W1031 23:05:51.418460 35074 ts_tablet_manager.cc:787] T b4e182b626554f3597d5c733c80997e6 P 0f53a0d3ef7e44ebb0365c800752d5bd: Trace: 1031 23:05:22.527332 (+ 0us) service_pool.cc:143] Inserting onto call queue 1031 23:05:22.527355 (+ 23us) service_pool.cc:202] Handling call 1031 23:05:22.527496 (+ 141us) ts_tablet_manager.cc:275] Acquired tablet manager lock 1031 23:05:22.527499 (+ 3us) ts_tablet_manager.cc:288] Creating new metadata... 1031 23:05:22.606117 (+ 78618us) tablet_metadata.cc:461] Metadata flushed 1031 23:05:22.607139 (+ 1022us) inbound_call.cc:130] Queueing success response 1031 23:05:22.607179 (+ 40us) ts_tablet_manager.cc:724] Bootstrapping tablet 1031 23:05:22.609040 (+ 1861us) ts_tablet_manager.cc:756] Initializing tablet peer 1031 23:05:22.609057 (+ 17us) tablet_peer.cc:161] Creating consensus instance 1031 23:05:22.609369 (+ 312us) tablet_peer.cc:183] Starting instrumentation 1031 23:05:22.609373 (+ 4us) tablet_peer.cc:188] TabletPeer::Init() finished 1031 23:05:22.609373 (+ 0us) ts_tablet_manager.cc:771] Starting tablet peer 1031 23:05:22.609374 (+ 1us) tablet_peer.cc:195] Starting consensus Metrics: {"fdatasync":1,"fdatasync_us":36157,"mutex_wait_us":28808186,"raft.queue_time_us":140,"raft.run_cpu_time_us":4,"raft.run_wall_time_us":4,"tablet-open.queue_time_us":527,"thread_start_us":1058,"threads_started":3} I1031 23:05:51.423633 5853 ts_tablet_manager.cc:1020] T 6a813ca355f749b6b31a773aa0f2ac7a P 0f53a0d3ef7e44ebb0365c800752d5bd: Deleting tablet data with delete state TABLET_DATA_DELETED I1031 23:05:51.525542 5853 ts_tablet_manager.cc:1032] T 6a813ca355f749b6b31a773aa0f2ac7a P 0f53a0d3ef7e44ebb0365c800752d5bd: Tablet deleted. Last logged OpId: 0.0 I1031 23:05:51.525622 5853 log.cc:876] T 6a813ca355f749b6b31a773aa0f2ac7a P 0f53a0d3ef7e44ebb0365c800752d5bd: Deleting WAL directory at /home/work/app/kudu/c3prc-kernel65/kudu_tablet_server/hdd1/wals/6a813ca355f749b6b31a773aa0f2ac7a I1031 23:05:51.525993 5853 consensus_meta.cc:71] T 6a813ca355f749b6b31a773aa0f2ac7a Deleting consensus metadata W1031 23:05:51.526253 5853 rpcz_store.cc:234] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 10.108.187.10:53997 (request call id 8) took 89049ms (client timeout 29999). W1031 23:05:51.526424 5853 rpcz_store.cc:238] Trace: 1031 23:04:22.476626 (+ 0us) service_pool.cc:143] Inserting onto call queue 1031 23:04:22.476656 (+ 30us) service_pool.cc:202] Handling call 1031 23:04:22.476709 (+ 53us) ts_tablet_manager.cc:572] Deleting tablet 6a813ca355f749b6b31a773aa0f2ac7a 1031 23:04:22.476709 (+ 0us) ts_tablet_manager.cc:580] Acquired tablet manager lock 1031 23:05:51.475074 (+88998365us) tablet_metadata.cc:461] Metadata flushed 1031 23:05:51.525516 (+ 50442us) tablet_metadata.cc:461] Metadata flushed 1031 23:05:51.526238 (+ 722us) inbound_call.cc:130] Queueing success response Metrics: {"fdatasync":2,"fdatasync_us":58254,"mutex_wait_us":88941266} W1031 23:05:51.583737 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 177 times. W1031 23:05:51.599557 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 237 times. I1031 23:05:51.720386 34574 raft_consensus.cc:411] T bff7dab34ccd4b42842077e82106f6fb P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I1031 23:05:51.720422 34574 raft_consensus.cc:2162] T bff7dab34ccd4b42842077e82106f6fb P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 11.457s I1031 23:05:51.720480 34574 raft_consensus.cc:435] T bff7dab34ccd4b42842077e82106f6fb P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f53a0d3ef7e44ebb0365c800752d5bd" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st01.bj" port: 18700 } } peers { permanent_uuid: "23f962e4a1744381ad5fa0d2d8b10241" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st07.bj" port: 18700 } } peers { permanent_uuid: "193181be29284287b25df2ee92938f0b" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st08.bj" port: 18700 } } I1031 23:05:51.721124 34574 leader_election.cc:216] T bff7dab34ccd4b42842077e82106f6fb P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 23f962e4a1744381ad5fa0d2d8b10241 I1031 23:05:51.721158 34574 leader_election.cc:216] T bff7dab34ccd4b42842077e82106f6fb P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 193181be29284287b25df2ee92938f0b W1031 23:05:51.721449 5816 leader_election.cc:278] T bff7dab34ccd4b42842077e82106f6fb P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 23f962e4a1744381ad5fa0d2d8b10241: Illegal state: Tablet not RUNNING: NOT_STARTED W1031 23:05:51.721513 5815 leader_election.cc:278] T bff7dab34ccd4b42842077e82106f6fb P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 193181be29284287b25df2ee92938f0b: Illegal state: Tablet not RUNNING: NOT_STARTED I1031 23:05:51.721563 5815 leader_election.cc:243] T bff7dab34ccd4b42842077e82106f6fb P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. I1031 23:05:51.721724 35208 raft_consensus.cc:2162] T bff7dab34ccd4b42842077e82106f6fb P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 12.826s I1031 23:05:51.721740 35208 raft_consensus.cc:1908] T bff7dab34ccd4b42842077e82106f6fb P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority I1031 23:05:51.749229 5734 maintenance_manager.cc:419] Time spent running UndoDeltaBlockGCOp(c62e1dbe04164d22bb6fdf7674bc18c5): real 0.332s user 0.153s sys 0.003s I1031 23:05:51.749482 5734 maintenance_manager.cc:425] P 0f53a0d3ef7e44ebb0365c800752d5bd: UndoDeltaBlockGCOp(c62e1dbe04164d22bb6fdf7674bc18c5) metrics: {"cfile_init":1,"data dir 6.queue_time_us":225,"data dir 6.run_cpu_time_us":113,"data dir 6.run_wall_time_us":9641,"fdatasync":1,"fdatasync_us":152695,"lbm_read_time_us":13,"lbm_reads_lt_1ms":4,"tcmalloc_contention_cycles":21504,"thread_start_us":212,"threads_started":1} W1031 23:05:51.814734 5815 consensus_peers.cc:357] T b4e182b626554f3597d5c733c80997e6 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 193181be29284287b25df2ee92938f0b (c3-kudu-tst-st08.bj:18700): Couldn't send request to peer 193181be29284287b25df2ee92938f0b for tablet b4e182b626554f3597d5c733c80997e6. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 58 times. W1031 23:05:52.083817 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 178 times. W1031 23:05:52.099637 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 238 times. I1031 23:05:52.277129 5734 maintenance_manager.cc:419] Time spent running UndoDeltaBlockGCOp(fffdb462a7184409ae79f01145545574): real 0.206s user 0.148s sys 0.003s I1031 23:05:52.277439 5734 maintenance_manager.cc:425] P 0f53a0d3ef7e44ebb0365c800752d5bd: UndoDeltaBlockGCOp(fffdb462a7184409ae79f01145545574) metrics: {"data dir 8.queue_time_us":232,"data dir 8.run_cpu_time_us":56,"data dir 8.run_wall_time_us":56,"fdatasync":1,"fdatasync_us":38955,"tcmalloc_contention_cycles":300032,"thread_start_us":219,"threads_started":1} W1031 23:05:52.314810 5815 consensus_peers.cc:357] T b4e182b626554f3597d5c733c80997e6 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 193181be29284287b25df2ee92938f0b (c3-kudu-tst-st08.bj:18700): Couldn't send request to peer 193181be29284287b25df2ee92938f0b for tablet b4e182b626554f3597d5c733c80997e6. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 59 times. I1031 23:05:52.547763 5853 tablet_service.cc:691] Processing DeleteTablet for tablet 012b21c0f95e4dd594076018fd6fdb7e with delete_type TABLET_DATA_DELETED (Replaced by ae7fa26c77054ef1b98d5e0f4e6940d6 at 2017-10-31 23:05:52 CST) from {username='work'} at 10.108.187.10:53997 I1031 23:05:52.547816 5853 tablet_peer.cc:221] Initiating TabletPeer shutdown for tablet: 012b21c0f95e4dd594076018fd6fdb7e I1031 23:05:52.547835 5853 maintenance_manager.cc:208] P 0f53a0d3ef7e44ebb0365c800752d5bd: Unregistered op CompactRowSetsOp(012b21c0f95e4dd594076018fd6fdb7e) I1031 23:05:52.547852 5853 maintenance_manager.cc:208] P 0f53a0d3ef7e44ebb0365c800752d5bd: Unregistered op MinorDeltaCompactionOp(012b21c0f95e4dd594076018fd6fdb7e) I1031 23:05:52.547864 5853 maintenance_manager.cc:208] P 0f53a0d3ef7e44ebb0365c800752d5bd: Unregistered op MajorDeltaCompactionOp(012b21c0f95e4dd594076018fd6fdb7e) I1031 23:05:52.547880 5853 maintenance_manager.cc:208] P 0f53a0d3ef7e44ebb0365c800752d5bd: Unregistered op UndoDeltaBlockGCOp(012b21c0f95e4dd594076018fd6fdb7e) I1031 23:05:52.547894 5853 maintenance_manager.cc:208] P 0f53a0d3ef7e44ebb0365c800752d5bd: Unregistered op FlushMRSOp(012b21c0f95e4dd594076018fd6fdb7e) I1031 23:05:52.547904 5853 maintenance_manager.cc:208] P 0f53a0d3ef7e44ebb0365c800752d5bd: Unregistered op FlushDeltaMemStoresOp(012b21c0f95e4dd594076018fd6fdb7e) I1031 23:05:52.547914 5853 maintenance_manager.cc:208] P 0f53a0d3ef7e44ebb0365c800752d5bd: Unregistered op LogGCOp(012b21c0f95e4dd594076018fd6fdb7e) I1031 23:05:52.547938 5853 raft_consensus.cc:1569] T 012b21c0f95e4dd594076018fd6fdb7e P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Raft consensus shutting down. I1031 23:05:52.547958 5853 raft_consensus.cc:1585] T 012b21c0f95e4dd594076018fd6fdb7e P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Raft consensus is shut down! I1031 23:05:52.551144 5853 ts_tablet_manager.cc:1020] T 012b21c0f95e4dd594076018fd6fdb7e P 0f53a0d3ef7e44ebb0365c800752d5bd: Deleting tablet data with delete state TABLET_DATA_DELETED W1031 23:05:52.583776 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 179 times. W1031 23:05:52.599609 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 239 times. I1031 23:05:52.702509 5853 ts_tablet_manager.cc:1032] T 012b21c0f95e4dd594076018fd6fdb7e P 0f53a0d3ef7e44ebb0365c800752d5bd: Tablet deleted. Last logged OpId: 0.0 I1031 23:05:52.702698 5853 log.cc:876] T 012b21c0f95e4dd594076018fd6fdb7e P 0f53a0d3ef7e44ebb0365c800752d5bd: Deleting WAL directory at /home/work/app/kudu/c3prc-kernel65/kudu_tablet_server/hdd1/wals/012b21c0f95e4dd594076018fd6fdb7e I1031 23:05:52.703032 5853 consensus_meta.cc:71] T 012b21c0f95e4dd594076018fd6fdb7e Deleting consensus metadata I1031 23:05:52.719094 5734 maintenance_manager.cc:419] Time spent running UndoDeltaBlockGCOp(fa1ead2de6944ab8ac2a26ea6bd67313): real 0.247s user 0.150s sys 0.004s I1031 23:05:52.719372 5734 maintenance_manager.cc:425] P 0f53a0d3ef7e44ebb0365c800752d5bd: UndoDeltaBlockGCOp(fa1ead2de6944ab8ac2a26ea6bd67313) metrics: {"data dir 7.queue_time_us":252,"data dir 7.run_cpu_time_us":74,"data dir 7.run_wall_time_us":75,"fdatasync":1,"fdatasync_us":61522,"tcmalloc_contention_cycles":48128,"thread_start_us":235,"threads_started":1} W1031 23:05:52.814913 5815 consensus_peers.cc:357] T b4e182b626554f3597d5c733c80997e6 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 193181be29284287b25df2ee92938f0b (c3-kudu-tst-st08.bj:18700): Couldn't send request to peer 193181be29284287b25df2ee92938f0b for tablet b4e182b626554f3597d5c733c80997e6. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 60 times. W1031 23:05:53.083971 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 180 times. I1031 23:05:53.086273 5734 maintenance_manager.cc:419] Time spent running UndoDeltaBlockGCOp(ee1d4d81f7b944c3884ec0ac9aa1373f): real 0.213s user 0.151s sys 0.003s I1031 23:05:53.086549 5734 maintenance_manager.cc:425] P 0f53a0d3ef7e44ebb0365c800752d5bd: UndoDeltaBlockGCOp(ee1d4d81f7b944c3884ec0ac9aa1373f) metrics: {"cfile_init":1,"data dir 6.queue_time_us":222,"data dir 6.run_cpu_time_us":55,"data dir 6.run_wall_time_us":56,"fdatasync":1,"fdatasync_us":35054,"lbm_read_time_us":13,"lbm_reads_lt_1ms":4,"tcmalloc_contention_cycles":97792,"thread_start_us":214,"threads_started":1} W1031 23:05:53.099752 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 240 times. I1031 23:05:53.289726 5734 maintenance_manager.cc:419] Time spent running UndoDeltaBlockGCOp(2605bb9cd2fe47fea1ad25d47ab8741d): real 0.016s user 0.015s sys 0.000s I1031 23:05:53.289764 5734 maintenance_manager.cc:425] P 0f53a0d3ef7e44ebb0365c800752d5bd: UndoDeltaBlockGCOp(2605bb9cd2fe47fea1ad25d47ab8741d) metrics: {"cfile_init":1,"lbm_read_time_us":13,"lbm_reads_lt_1ms":4} W1031 23:05:53.314927 5815 consensus_peers.cc:357] T b4e182b626554f3597d5c733c80997e6 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 193181be29284287b25df2ee92938f0b (c3-kudu-tst-st08.bj:18700): Couldn't send request to peer 193181be29284287b25df2ee92938f0b for tablet b4e182b626554f3597d5c733c80997e6. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 61 times. W1031 23:05:53.583894 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 181 times. W1031 23:05:53.599756 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 241 times. W1031 23:05:53.815044 5815 consensus_peers.cc:357] T b4e182b626554f3597d5c733c80997e6 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 193181be29284287b25df2ee92938f0b (c3-kudu-tst-st08.bj:18700): Couldn't send request to peer 193181be29284287b25df2ee92938f0b for tablet b4e182b626554f3597d5c733c80997e6. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 62 times. I1031 23:05:53.907959 5734 tablet.cc:911] T 3eb1e56cad854272946b9a65dc35259b P 0f53a0d3ef7e44ebb0365c800752d5bd: Flush: entering stage 1 (old memrowset already frozen for inserts) I1031 23:05:53.907994 5734 compaction.cc:877] Selected 1 rowsets to compact: I1031 23:05:53.908006 5734 compaction.cc:880] memrowset(current size on disk: ~0 bytes) I1031 23:05:53.908022 5734 tablet.cc:913] T 3eb1e56cad854272946b9a65dc35259b P 0f53a0d3ef7e44ebb0365c800752d5bd: Memstore in-memory size: 3751 bytes I1031 23:05:53.908054 5734 tablet.cc:1174] T 3eb1e56cad854272946b9a65dc35259b P 0f53a0d3ef7e44ebb0365c800752d5bd: Flush: entering phase 1 (flushing snapshot). Phase 1 snapshot: MvccSnapshot[committed={T|T < 6182757722229637120 or (T in {6182757722229637120})}] I1031 23:05:53.908174 5734 multi_column_writer.cc:85] Opened CFile writer for column ts[int32 NOT NULL] I1031 23:05:53.908316 5734 multi_column_writer.cc:85] Opened CFile writer for column id[int32 NOT NULL] W1031 23:05:54.084130 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 182 times. W1031 23:05:54.099896 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 242 times. I1031 23:05:54.162991 5734 tablet.cc:1259] T 3eb1e56cad854272946b9a65dc35259b P 0f53a0d3ef7e44ebb0365c800752d5bd: Flush: entering phase 2 (starting to duplicate updates in new rowsets) W1031 23:05:54.315008 5815 consensus_peers.cc:357] T b4e182b626554f3597d5c733c80997e6 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 193181be29284287b25df2ee92938f0b (c3-kudu-tst-st08.bj:18700): Couldn't send request to peer 193181be29284287b25df2ee92938f0b for tablet b4e182b626554f3597d5c733c80997e6. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 63 times. I1031 23:05:54.390940 5734 tablet.cc:1314] T 3eb1e56cad854272946b9a65dc35259b P 0f53a0d3ef7e44ebb0365c800752d5bd: Flush Phase 2: carrying over any updates which arrived during Phase 1 I1031 23:05:54.391016 5734 tablet.cc:1316] T 3eb1e56cad854272946b9a65dc35259b P 0f53a0d3ef7e44ebb0365c800752d5bd: Phase 2 snapshot: MvccSnapshot[committed={T|T < 6182757722229637120 or (T in {6182757722229637120})}] W1031 23:05:54.584168 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 183 times. W1031 23:05:54.599925 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 243 times. W1031 23:05:54.815161 5815 consensus_peers.cc:357] T b4e182b626554f3597d5c733c80997e6 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 193181be29284287b25df2ee92938f0b (c3-kudu-tst-st08.bj:18700): Couldn't send request to peer 193181be29284287b25df2ee92938f0b for tablet b4e182b626554f3597d5c733c80997e6. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 64 times. I1031 23:05:54.865501 5734 tablet.cc:1358] T 3eb1e56cad854272946b9a65dc35259b P 0f53a0d3ef7e44ebb0365c800752d5bd: Flush successful on 10 rows (4330 bytes) I1031 23:05:54.865602 5734 maintenance_manager.cc:419] Time spent running FlushMRSOp(3eb1e56cad854272946b9a65dc35259b): real 1.189s user 0.815s sys 0.005s I1031 23:05:54.865768 5734 maintenance_manager.cc:425] P 0f53a0d3ef7e44ebb0365c800752d5bd: FlushMRSOp(3eb1e56cad854272946b9a65dc35259b) metrics: {"cfile_init":1,"fdatasync":11,"fdatasync_us":305890,"lbm_read_time_us":6,"lbm_reads_lt_1ms":4,"lbm_write_time_us":13613,"lbm_writes_10-100_ms":1,"lbm_writes_lt_1ms":25,"tcmalloc_contention_cycles":480896} W1031 23:05:55.084120 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 184 times. W1031 23:05:55.099998 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 244 times. W1031 23:05:55.315237 5815 consensus_peers.cc:357] T b4e182b626554f3597d5c733c80997e6 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 193181be29284287b25df2ee92938f0b (c3-kudu-tst-st08.bj:18700): Couldn't send request to peer 193181be29284287b25df2ee92938f0b for tablet b4e182b626554f3597d5c733c80997e6. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 65 times. W1031 23:05:55.584264 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 185 times. W1031 23:05:55.600177 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 245 times. W1031 23:05:55.815326 5815 consensus_peers.cc:357] T b4e182b626554f3597d5c733c80997e6 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 193181be29284287b25df2ee92938f0b (c3-kudu-tst-st08.bj:18700): Couldn't send request to peer 193181be29284287b25df2ee92938f0b for tablet b4e182b626554f3597d5c733c80997e6. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 66 times. W1031 23:05:56.084314 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 186 times. W1031 23:05:56.100261 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 246 times. W1031 23:05:56.315484 5815 consensus_peers.cc:357] T b4e182b626554f3597d5c733c80997e6 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 193181be29284287b25df2ee92938f0b (c3-kudu-tst-st08.bj:18700): Couldn't send request to peer 193181be29284287b25df2ee92938f0b for tablet b4e182b626554f3597d5c733c80997e6. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 67 times. W1031 23:05:56.584388 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 187 times. W1031 23:05:56.600329 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 247 times. W1031 23:05:56.815505 5815 consensus_peers.cc:357] T b4e182b626554f3597d5c733c80997e6 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 193181be29284287b25df2ee92938f0b (c3-kudu-tst-st08.bj:18700): Couldn't send request to peer 193181be29284287b25df2ee92938f0b for tablet b4e182b626554f3597d5c733c80997e6. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 68 times. W1031 23:05:57.084499 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 188 times. W1031 23:05:57.100380 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 248 times. W1031 23:05:57.315551 5815 consensus_peers.cc:357] T b4e182b626554f3597d5c733c80997e6 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 193181be29284287b25df2ee92938f0b (c3-kudu-tst-st08.bj:18700): Couldn't send request to peer 193181be29284287b25df2ee92938f0b for tablet b4e182b626554f3597d5c733c80997e6. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 69 times. W1031 23:05:57.584591 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 189 times. W1031 23:05:57.600430 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 249 times. W1031 23:05:57.815639 5815 consensus_peers.cc:357] T b4e182b626554f3597d5c733c80997e6 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 193181be29284287b25df2ee92938f0b (c3-kudu-tst-st08.bj:18700): Couldn't send request to peer 193181be29284287b25df2ee92938f0b for tablet b4e182b626554f3597d5c733c80997e6. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 70 times. I1031 23:05:57.965672 34590 raft_consensus.cc:411] T 117cd74bcdab4942901acb0005cdd702 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I1031 23:05:57.965723 34590 raft_consensus.cc:2162] T 117cd74bcdab4942901acb0005cdd702 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 11.263s I1031 23:05:57.965783 34590 raft_consensus.cc:435] T 117cd74bcdab4942901acb0005cdd702 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625f49b9554549e6a5c3a92b76e576cf" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st10.bj" port: 18700 } } peers { permanent_uuid: "0a8bbcbbb5a6459aa5258464755db3bb" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st02.bj" port: 18700 } } peers { permanent_uuid: "0f53a0d3ef7e44ebb0365c800752d5bd" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st01.bj" port: 18700 } } I1031 23:05:57.966501 34590 leader_election.cc:216] T 117cd74bcdab4942901acb0005cdd702 P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 625f49b9554549e6a5c3a92b76e576cf I1031 23:05:57.966557 34590 leader_election.cc:216] T 117cd74bcdab4942901acb0005cdd702 P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 0a8bbcbbb5a6459aa5258464755db3bb W1031 23:05:57.966758 5814 leader_election.cc:278] T 117cd74bcdab4942901acb0005cdd702 P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 625f49b9554549e6a5c3a92b76e576cf: Not found: Tablet not found W1031 23:05:57.966806 5816 leader_election.cc:278] T 117cd74bcdab4942901acb0005cdd702 P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0a8bbcbbb5a6459aa5258464755db3bb: Not found: Tablet not found I1031 23:05:57.966827 5816 leader_election.cc:243] T 117cd74bcdab4942901acb0005cdd702 P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. I1031 23:05:57.967083 35245 raft_consensus.cc:2162] T 117cd74bcdab4942901acb0005cdd702 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 13.600s I1031 23:05:57.967100 35245 raft_consensus.cc:1908] T 117cd74bcdab4942901acb0005cdd702 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority W1031 23:05:58.084529 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 190 times. W1031 23:05:58.100580 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 250 times. I1031 23:05:58.273815 5853 tablet_service.cc:691] Processing DeleteTablet for tablet 117cd74bcdab4942901acb0005cdd702 with delete_type TABLET_DATA_DELETED (Replaced by acb401e40e7049fbbfdbeb40474353d9 at 2017-10-31 23:04:52 CST) from {username='work'} at 10.108.187.10:53997 I1031 23:05:58.273851 5853 tablet_peer.cc:221] Initiating TabletPeer shutdown for tablet: 117cd74bcdab4942901acb0005cdd702 I1031 23:05:58.312867 5857 tablet_service.cc:691] Processing DeleteTablet for tablet 6a813ca355f749b6b31a773aa0f2ac7a with delete_type TABLET_DATA_DELETED (Replaced by 582c1281bf5147b2bd40c9a950a1152f at 2017-10-31 23:04:22 CST) from {username='work'} at 10.108.187.10:53997 W1031 23:05:58.315769 5815 consensus_peers.cc:357] T b4e182b626554f3597d5c733c80997e6 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 193181be29284287b25df2ee92938f0b (c3-kudu-tst-st08.bj:18700): Couldn't send request to peer 193181be29284287b25df2ee92938f0b for tablet b4e182b626554f3597d5c733c80997e6. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 71 times. I1031 23:05:58.330807 5857 tablet_service.cc:691] Processing DeleteTablet for tablet bff7dab34ccd4b42842077e82106f6fb with delete_type TABLET_DATA_DELETED (Replaced by 005a48fb466c4a52896842dd00124755 at 2017-10-31 23:04:52 CST) from {username='work'} at 10.108.187.10:53997 I1031 23:05:58.330828 5857 tablet_peer.cc:221] Initiating TabletPeer shutdown for tablet: bff7dab34ccd4b42842077e82106f6fb I1031 23:05:58.378032 5861 tablet_service.cc:691] Processing DeleteTablet for tablet 56155a2adc24450d89e338cc955bfa18 with delete_type TABLET_DATA_DELETED (Replaced by 80ba7f33ec864c5f96c61932edb5cdc4 at 2017-10-31 23:04:52 CST) from {username='work'} at 10.108.187.10:53997 I1031 23:05:58.378118 5861 tablet_peer.cc:221] Initiating TabletPeer shutdown for tablet: 56155a2adc24450d89e338cc955bfa18 W1031 23:05:58.584717 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 191 times. W1031 23:05:58.600558 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 251 times. W1031 23:05:58.815770 5815 consensus_peers.cc:357] T b4e182b626554f3597d5c733c80997e6 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 193181be29284287b25df2ee92938f0b (c3-kudu-tst-st08.bj:18700): Couldn't send request to peer 193181be29284287b25df2ee92938f0b for tablet b4e182b626554f3597d5c733c80997e6. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 72 times. W1031 23:05:59.084785 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 192 times. W1031 23:05:59.100683 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 252 times. W1031 23:05:59.315829 5815 consensus_peers.cc:357] T b4e182b626554f3597d5c733c80997e6 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 193181be29284287b25df2ee92938f0b (c3-kudu-tst-st08.bj:18700): Couldn't send request to peer 193181be29284287b25df2ee92938f0b for tablet b4e182b626554f3597d5c733c80997e6. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 73 times. W1031 23:05:59.584925 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 193 times. W1031 23:05:59.600733 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 253 times. W1031 23:05:59.815922 5815 consensus_peers.cc:357] T b4e182b626554f3597d5c733c80997e6 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 193181be29284287b25df2ee92938f0b (c3-kudu-tst-st08.bj:18700): Couldn't send request to peer 193181be29284287b25df2ee92938f0b for tablet b4e182b626554f3597d5c733c80997e6. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 74 times. W1031 23:06:00.084980 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 194 times. W1031 23:06:00.100865 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 254 times. W1031 23:06:00.316061 5815 consensus_peers.cc:357] T b4e182b626554f3597d5c733c80997e6 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 193181be29284287b25df2ee92938f0b (c3-kudu-tst-st08.bj:18700): Couldn't send request to peer 193181be29284287b25df2ee92938f0b for tablet b4e182b626554f3597d5c733c80997e6. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 75 times. W1031 23:06:00.585044 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 195 times. W1031 23:06:00.600894 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 255 times. W1031 23:06:00.816123 5815 consensus_peers.cc:357] T b4e182b626554f3597d5c733c80997e6 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 193181be29284287b25df2ee92938f0b (c3-kudu-tst-st08.bj:18700): Couldn't send request to peer 193181be29284287b25df2ee92938f0b for tablet b4e182b626554f3597d5c733c80997e6. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 76 times. W1031 23:06:01.085069 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 196 times. W1031 23:06:01.100998 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 256 times. W1031 23:06:01.316093 5815 consensus_peers.cc:357] T b4e182b626554f3597d5c733c80997e6 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 193181be29284287b25df2ee92938f0b (c3-kudu-tst-st08.bj:18700): Couldn't send request to peer 193181be29284287b25df2ee92938f0b for tablet b4e182b626554f3597d5c733c80997e6. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 77 times. W1031 23:06:01.585237 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 197 times. W1031 23:06:01.601052 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 257 times. W1031 23:06:01.816300 5815 consensus_peers.cc:357] T b4e182b626554f3597d5c733c80997e6 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 193181be29284287b25df2ee92938f0b (c3-kudu-tst-st08.bj:18700): Couldn't send request to peer 193181be29284287b25df2ee92938f0b for tablet b4e182b626554f3597d5c733c80997e6. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 78 times. W1031 23:06:02.085232 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 198 times. W1031 23:06:02.101053 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 258 times. I1031 23:06:02.317131 35078 consensus_queue.cc:695] T b4e182b626554f3597d5c733c80997e6 P 0f53a0d3ef7e44ebb0365c800752d5bd [LEADER]: Connected to new peer: Peer: 193181be29284287b25df2ee92938f0b, Is new: false, Last received: 0.0, Next index: 1, Last known committed idx: 0, Last exchange result: ERROR, Needs tablet copy: false W1031 23:06:02.585399 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 199 times. W1031 23:06:02.601125 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 259 times. W1031 23:06:03.085460 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 200 times. W1031 23:06:03.101227 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 260 times. W1031 23:06:03.585474 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 201 times. W1031 23:06:03.601238 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 261 times. I1031 23:06:03.881225 34585 raft_consensus.cc:411] T 56155a2adc24450d89e338cc955bfa18 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I1031 23:06:03.881271 34585 raft_consensus.cc:2162] T 56155a2adc24450d89e338cc955bfa18 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 2.556s I1031 23:06:03.881337 34585 raft_consensus.cc:435] T 56155a2adc24450d89e338cc955bfa18 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625f49b9554549e6a5c3a92b76e576cf" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st10.bj" port: 18700 } } peers { permanent_uuid: "193181be29284287b25df2ee92938f0b" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st08.bj" port: 18700 } } peers { permanent_uuid: "0f53a0d3ef7e44ebb0365c800752d5bd" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st01.bj" port: 18700 } } I1031 23:06:03.881886 34585 leader_election.cc:216] T 56155a2adc24450d89e338cc955bfa18 P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 625f49b9554549e6a5c3a92b76e576cf I1031 23:06:03.881925 34585 leader_election.cc:216] T 56155a2adc24450d89e338cc955bfa18 P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 193181be29284287b25df2ee92938f0b W1031 23:06:03.882274 5814 leader_election.cc:278] T 56155a2adc24450d89e338cc955bfa18 P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 625f49b9554549e6a5c3a92b76e576cf: Not found: Tablet not found W1031 23:06:03.882340 5815 leader_election.cc:278] T 56155a2adc24450d89e338cc955bfa18 P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 193181be29284287b25df2ee92938f0b: Not found: Tablet not found I1031 23:06:03.882359 5815 leader_election.cc:243] T 56155a2adc24450d89e338cc955bfa18 P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. I1031 23:06:03.882707 35266 raft_consensus.cc:2162] T 56155a2adc24450d89e338cc955bfa18 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 17.378s I1031 23:06:03.882730 35266 raft_consensus.cc:1908] T 56155a2adc24450d89e338cc955bfa18 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority W1031 23:06:04.085563 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 202 times. W1031 23:06:04.101380 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 262 times. W1031 23:06:04.585649 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 203 times. W1031 23:06:04.601469 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 263 times. W1031 23:06:05.085772 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 204 times. W1031 23:06:05.101557 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 264 times. W1031 23:06:05.585764 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 205 times. W1031 23:06:05.601569 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 265 times. W1031 23:06:06.085881 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 206 times. W1031 23:06:06.101773 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 266 times. I1031 23:06:06.355962 34574 raft_consensus.cc:411] T bff7dab34ccd4b42842077e82106f6fb P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I1031 23:06:06.356003 34574 raft_consensus.cc:2162] T bff7dab34ccd4b42842077e82106f6fb P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 4.211s I1031 23:06:06.356053 34574 raft_consensus.cc:435] T bff7dab34ccd4b42842077e82106f6fb P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f53a0d3ef7e44ebb0365c800752d5bd" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st01.bj" port: 18700 } } peers { permanent_uuid: "23f962e4a1744381ad5fa0d2d8b10241" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st07.bj" port: 18700 } } peers { permanent_uuid: "193181be29284287b25df2ee92938f0b" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st08.bj" port: 18700 } } I1031 23:06:06.356572 34574 leader_election.cc:216] T bff7dab34ccd4b42842077e82106f6fb P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 23f962e4a1744381ad5fa0d2d8b10241 I1031 23:06:06.356604 34574 leader_election.cc:216] T bff7dab34ccd4b42842077e82106f6fb P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 193181be29284287b25df2ee92938f0b W1031 23:06:06.356863 5816 leader_election.cc:278] T bff7dab34ccd4b42842077e82106f6fb P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 23f962e4a1744381ad5fa0d2d8b10241: Illegal state: Tablet not RUNNING: NOT_STARTED W1031 23:06:06.356993 5815 leader_election.cc:278] T bff7dab34ccd4b42842077e82106f6fb P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 193181be29284287b25df2ee92938f0b: Not found: Tablet not found I1031 23:06:06.357034 5815 leader_election.cc:243] T bff7dab34ccd4b42842077e82106f6fb P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. I1031 23:06:06.357287 35287 raft_consensus.cc:2162] T bff7dab34ccd4b42842077e82106f6fb P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 13.706s I1031 23:06:06.357308 35287 raft_consensus.cc:1908] T bff7dab34ccd4b42842077e82106f6fb P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority W1031 23:06:06.586076 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 207 times. W1031 23:06:06.601856 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 267 times. W1031 23:06:07.086120 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 208 times. W1031 23:06:07.101943 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 268 times. W1031 23:06:07.586072 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 209 times. W1031 23:06:07.602058 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 269 times. W1031 23:06:08.086257 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 210 times. W1031 23:06:08.102128 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 270 times. W1031 23:06:08.586230 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 211 times. W1031 23:06:08.602176 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 271 times. W1031 23:06:09.086410 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 212 times. W1031 23:06:09.102188 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 272 times. W1031 23:06:09.586462 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 213 times. W1031 23:06:09.602249 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 273 times. W1031 23:06:10.086575 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 214 times. W1031 23:06:10.102346 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 274 times. W1031 23:06:10.586570 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 215 times. W1031 23:06:10.602314 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 275 times. W1031 23:06:11.086715 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 216 times. W1031 23:06:11.102511 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 276 times. W1031 23:06:11.586776 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 217 times. W1031 23:06:11.602504 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 277 times. W1031 23:06:12.086844 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 218 times. W1031 23:06:12.102557 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 278 times. W1031 23:06:12.586884 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 219 times. W1031 23:06:12.602660 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 279 times. W1031 23:06:13.087034 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 220 times. W1031 23:06:13.102776 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 280 times. I1031 23:06:13.322135 34590 raft_consensus.cc:411] T 117cd74bcdab4942901acb0005cdd702 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I1031 23:06:13.322209 34590 raft_consensus.cc:2162] T 117cd74bcdab4942901acb0005cdd702 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 16.372s I1031 23:06:13.322286 34590 raft_consensus.cc:435] T 117cd74bcdab4942901acb0005cdd702 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625f49b9554549e6a5c3a92b76e576cf" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st10.bj" port: 18700 } } peers { permanent_uuid: "0a8bbcbbb5a6459aa5258464755db3bb" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st02.bj" port: 18700 } } peers { permanent_uuid: "0f53a0d3ef7e44ebb0365c800752d5bd" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st01.bj" port: 18700 } } I1031 23:06:13.322854 34590 leader_election.cc:216] T 117cd74bcdab4942901acb0005cdd702 P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 625f49b9554549e6a5c3a92b76e576cf I1031 23:06:13.322901 34590 leader_election.cc:216] T 117cd74bcdab4942901acb0005cdd702 P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 0a8bbcbbb5a6459aa5258464755db3bb W1031 23:06:13.323168 5816 leader_election.cc:278] T 117cd74bcdab4942901acb0005cdd702 P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0a8bbcbbb5a6459aa5258464755db3bb: Not found: Tablet not found W1031 23:06:13.323254 5814 leader_election.cc:278] T 117cd74bcdab4942901acb0005cdd702 P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 625f49b9554549e6a5c3a92b76e576cf: Not found: Tablet not found I1031 23:06:13.323287 5814 leader_election.cc:243] T 117cd74bcdab4942901acb0005cdd702 P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. I1031 23:06:13.323524 35318 raft_consensus.cc:2162] T 117cd74bcdab4942901acb0005cdd702 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 14.854s I1031 23:06:13.323546 35318 raft_consensus.cc:1908] T 117cd74bcdab4942901acb0005cdd702 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority W1031 23:06:13.587069 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 221 times. W1031 23:06:13.602792 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 281 times. W1031 23:06:14.087072 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 222 times. W1031 23:06:14.102823 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 282 times. W1031 23:06:14.587103 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 223 times. W1031 23:06:14.603034 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 283 times. W1031 23:06:15.087244 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 224 times. W1031 23:06:15.103117 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 284 times. W1031 23:06:15.587196 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 225 times. W1031 23:06:15.603088 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 285 times. W1031 23:06:16.087388 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 226 times. W1031 23:06:16.103155 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 286 times. W1031 23:06:16.587409 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 227 times. W1031 23:06:16.603325 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 287 times. W1031 23:06:17.087407 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 228 times. W1031 23:06:17.103356 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 288 times. W1031 23:06:17.587548 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 229 times. W1031 23:06:17.603425 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 289 times. W1031 23:06:18.087661 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 230 times. W1031 23:06:18.103411 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 290 times. W1031 23:06:18.587707 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 231 times. W1031 23:06:18.603546 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 291 times. W1031 23:06:19.087813 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 232 times. W1031 23:06:19.103583 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 292 times. W1031 23:06:19.587929 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 233 times. W1031 23:06:19.603683 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 293 times. W1031 23:06:20.088019 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 234 times. W1031 23:06:20.103744 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 294 times. W1031 23:06:20.587993 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 235 times. W1031 23:06:20.603907 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 295 times. W1031 23:06:21.088006 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 236 times. W1031 23:06:21.103991 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 296 times. W1031 23:06:21.588089 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 237 times. W1031 23:06:21.603940 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 297 times. I1031 23:06:21.643834 34574 raft_consensus.cc:411] T bff7dab34ccd4b42842077e82106f6fb P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I1031 23:06:21.643874 34574 raft_consensus.cc:2162] T bff7dab34ccd4b42842077e82106f6fb P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 12.753s I1031 23:06:21.643939 34574 raft_consensus.cc:435] T bff7dab34ccd4b42842077e82106f6fb P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f53a0d3ef7e44ebb0365c800752d5bd" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st01.bj" port: 18700 } } peers { permanent_uuid: "23f962e4a1744381ad5fa0d2d8b10241" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st07.bj" port: 18700 } } peers { permanent_uuid: "193181be29284287b25df2ee92938f0b" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st08.bj" port: 18700 } } I1031 23:06:21.644428 34574 leader_election.cc:216] T bff7dab34ccd4b42842077e82106f6fb P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 23f962e4a1744381ad5fa0d2d8b10241 I1031 23:06:21.644456 34574 leader_election.cc:216] T bff7dab34ccd4b42842077e82106f6fb P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 193181be29284287b25df2ee92938f0b W1031 23:06:21.644681 5816 leader_election.cc:278] T bff7dab34ccd4b42842077e82106f6fb P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 23f962e4a1744381ad5fa0d2d8b10241: Illegal state: Tablet not RUNNING: NOT_STARTED W1031 23:06:21.644810 5815 leader_election.cc:278] T bff7dab34ccd4b42842077e82106f6fb P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 193181be29284287b25df2ee92938f0b: Not found: Tablet not found I1031 23:06:21.644842 5815 leader_election.cc:243] T bff7dab34ccd4b42842077e82106f6fb P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. I1031 23:06:21.645092 35557 raft_consensus.cc:2162] T bff7dab34ccd4b42842077e82106f6fb P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 15.466s I1031 23:06:21.645112 35557 raft_consensus.cc:1908] T bff7dab34ccd4b42842077e82106f6fb P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority W1031 23:06:22.088327 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 238 times. W1031 23:06:22.104158 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 298 times. W1031 23:06:22.588336 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 239 times. W1031 23:06:22.604245 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 299 times. I1031 23:06:23.071887 34585 raft_consensus.cc:411] T 56155a2adc24450d89e338cc955bfa18 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I1031 23:06:23.071938 34585 raft_consensus.cc:2162] T 56155a2adc24450d89e338cc955bfa18 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 10.929s I1031 23:06:23.072001 34585 raft_consensus.cc:435] T 56155a2adc24450d89e338cc955bfa18 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625f49b9554549e6a5c3a92b76e576cf" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st10.bj" port: 18700 } } peers { permanent_uuid: "193181be29284287b25df2ee92938f0b" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st08.bj" port: 18700 } } peers { permanent_uuid: "0f53a0d3ef7e44ebb0365c800752d5bd" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st01.bj" port: 18700 } } I1031 23:06:23.072646 34585 leader_election.cc:216] T 56155a2adc24450d89e338cc955bfa18 P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 625f49b9554549e6a5c3a92b76e576cf I1031 23:06:23.072685 34585 leader_election.cc:216] T 56155a2adc24450d89e338cc955bfa18 P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 193181be29284287b25df2ee92938f0b W1031 23:06:23.072873 5814 leader_election.cc:278] T 56155a2adc24450d89e338cc955bfa18 P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 625f49b9554549e6a5c3a92b76e576cf: Not found: Tablet not found W1031 23:06:23.072964 5815 leader_election.cc:278] T 56155a2adc24450d89e338cc955bfa18 P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 193181be29284287b25df2ee92938f0b: Not found: Tablet not found I1031 23:06:23.072988 5815 leader_election.cc:243] T 56155a2adc24450d89e338cc955bfa18 P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. I1031 23:06:23.073235 35560 raft_consensus.cc:2162] T 56155a2adc24450d89e338cc955bfa18 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 13.584s I1031 23:06:23.073253 35560 raft_consensus.cc:1908] T 56155a2adc24450d89e338cc955bfa18 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority W1031 23:06:23.088409 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 240 times. W1031 23:06:23.104202 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 300 times. W1031 23:06:23.588496 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 241 times. W1031 23:06:23.604280 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 301 times. W1031 23:06:24.088624 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 242 times. W1031 23:06:24.104410 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 302 times. W1031 23:06:24.588663 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 243 times. W1031 23:06:24.604537 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 303 times. W1031 23:06:25.088881 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 244 times. W1031 23:06:25.104638 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 304 times. W1031 23:06:25.588806 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 245 times. W1031 23:06:25.604593 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 305 times. W1031 23:06:26.088922 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 246 times. W1031 23:06:26.104774 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 306 times. W1031 23:06:26.588930 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 247 times. W1031 23:06:26.604845 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 307 times. W1031 23:06:27.089077 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 248 times. W1031 23:06:27.104892 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 308 times. W1031 23:06:27.589072 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 249 times. W1031 23:06:27.605000 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 309 times. W1031 23:06:28.089272 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 250 times. W1031 23:06:28.105119 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 310 times. W1031 23:06:28.589283 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 251 times. W1031 23:06:28.605068 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 311 times. W1031 23:06:29.089399 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 252 times. W1031 23:06:29.105080 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 312 times. W1031 23:06:29.589355 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 253 times. W1031 23:06:29.605219 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 313 times. W1031 23:06:30.089593 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 254 times. W1031 23:06:30.105350 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 314 times. I1031 23:06:30.230433 34590 raft_consensus.cc:411] T 117cd74bcdab4942901acb0005cdd702 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I1031 23:06:30.230474 34590 raft_consensus.cc:2162] T 117cd74bcdab4942901acb0005cdd702 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 13.943s I1031 23:06:30.230528 34590 raft_consensus.cc:435] T 117cd74bcdab4942901acb0005cdd702 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625f49b9554549e6a5c3a92b76e576cf" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st10.bj" port: 18700 } } peers { permanent_uuid: "0a8bbcbbb5a6459aa5258464755db3bb" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st02.bj" port: 18700 } } peers { permanent_uuid: "0f53a0d3ef7e44ebb0365c800752d5bd" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st01.bj" port: 18700 } } I1031 23:06:30.231083 34590 leader_election.cc:216] T 117cd74bcdab4942901acb0005cdd702 P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 625f49b9554549e6a5c3a92b76e576cf I1031 23:06:30.231109 34590 leader_election.cc:216] T 117cd74bcdab4942901acb0005cdd702 P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 0a8bbcbbb5a6459aa5258464755db3bb W1031 23:06:30.231320 5816 leader_election.cc:278] T 117cd74bcdab4942901acb0005cdd702 P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0a8bbcbbb5a6459aa5258464755db3bb: Not found: Tablet not found W1031 23:06:30.231472 5814 leader_election.cc:278] T 117cd74bcdab4942901acb0005cdd702 P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 625f49b9554549e6a5c3a92b76e576cf: Not found: Tablet not found I1031 23:06:30.231509 5814 leader_election.cc:243] T 117cd74bcdab4942901acb0005cdd702 P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. I1031 23:06:30.231771 35588 raft_consensus.cc:2162] T 117cd74bcdab4942901acb0005cdd702 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 7.763s I1031 23:06:30.231793 35588 raft_consensus.cc:1908] T 117cd74bcdab4942901acb0005cdd702 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority W1031 23:06:30.589507 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 255 times. W1031 23:06:30.605376 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 315 times. W1031 23:06:31.089625 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 256 times. W1031 23:06:31.105530 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 316 times. W1031 23:06:31.589664 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 257 times. W1031 23:06:31.605504 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 317 times. W1031 23:06:32.089782 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 258 times. W1031 23:06:32.105619 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 318 times. W1031 23:06:32.589810 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 259 times. W1031 23:06:32.605756 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 319 times. W1031 23:06:33.089946 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 260 times. W1031 23:06:33.105808 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 320 times. W1031 23:06:33.589964 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 261 times. W1031 23:06:33.605819 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 321 times. W1031 23:06:34.090131 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 262 times. W1031 23:06:34.105952 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 322 times. W1031 23:06:34.590175 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 263 times. W1031 23:06:34.606070 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 323 times. W1031 23:06:35.090245 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 264 times. W1031 23:06:35.106135 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 324 times. W1031 23:06:35.590291 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 265 times. W1031 23:06:35.606241 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 325 times. W1031 23:06:36.090463 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 266 times. W1031 23:06:36.106343 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 326 times. W1031 23:06:36.590503 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 267 times. W1031 23:06:36.606420 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 327 times. W1031 23:06:37.090595 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 268 times. W1031 23:06:37.106487 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 328 times. W1031 23:06:37.590657 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 269 times. W1031 23:06:37.606420 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 329 times. W1031 23:06:38.090796 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 270 times. W1031 23:06:38.106593 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 330 times. I1031 23:06:38.465358 34585 raft_consensus.cc:411] T 56155a2adc24450d89e338cc955bfa18 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I1031 23:06:38.465411 34585 raft_consensus.cc:2162] T 56155a2adc24450d89e338cc955bfa18 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 12.871s I1031 23:06:38.465467 34585 raft_consensus.cc:435] T 56155a2adc24450d89e338cc955bfa18 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625f49b9554549e6a5c3a92b76e576cf" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st10.bj" port: 18700 } } peers { permanent_uuid: "193181be29284287b25df2ee92938f0b" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st08.bj" port: 18700 } } peers { permanent_uuid: "0f53a0d3ef7e44ebb0365c800752d5bd" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st01.bj" port: 18700 } } I1031 23:06:38.466023 34585 leader_election.cc:216] T 56155a2adc24450d89e338cc955bfa18 P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 625f49b9554549e6a5c3a92b76e576cf I1031 23:06:38.466063 34585 leader_election.cc:216] T 56155a2adc24450d89e338cc955bfa18 P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 193181be29284287b25df2ee92938f0b W1031 23:06:38.466388 5814 leader_election.cc:278] T 56155a2adc24450d89e338cc955bfa18 P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 625f49b9554549e6a5c3a92b76e576cf: Not found: Tablet not found W1031 23:06:38.466439 5815 leader_election.cc:278] T 56155a2adc24450d89e338cc955bfa18 P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 193181be29284287b25df2ee92938f0b: Not found: Tablet not found I1031 23:06:38.466457 5815 leader_election.cc:243] T 56155a2adc24450d89e338cc955bfa18 P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. I1031 23:06:38.466730 35625 raft_consensus.cc:2162] T 56155a2adc24450d89e338cc955bfa18 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 2.504s I1031 23:06:38.466753 35625 raft_consensus.cc:1908] T 56155a2adc24450d89e338cc955bfa18 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority W1031 23:06:38.590868 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 271 times. W1031 23:06:38.606608 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 331 times. I1031 23:06:38.964960 34574 raft_consensus.cc:411] T bff7dab34ccd4b42842077e82106f6fb P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I1031 23:06:38.965003 34574 raft_consensus.cc:2162] T bff7dab34ccd4b42842077e82106f6fb P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 13.681s I1031 23:06:38.965057 34574 raft_consensus.cc:435] T bff7dab34ccd4b42842077e82106f6fb P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f53a0d3ef7e44ebb0365c800752d5bd" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st01.bj" port: 18700 } } peers { permanent_uuid: "23f962e4a1744381ad5fa0d2d8b10241" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st07.bj" port: 18700 } } peers { permanent_uuid: "193181be29284287b25df2ee92938f0b" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st08.bj" port: 18700 } } I1031 23:06:38.965601 34574 leader_election.cc:216] T bff7dab34ccd4b42842077e82106f6fb P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 23f962e4a1744381ad5fa0d2d8b10241 I1031 23:06:38.965638 34574 leader_election.cc:216] T bff7dab34ccd4b42842077e82106f6fb P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 193181be29284287b25df2ee92938f0b W1031 23:06:38.965894 5816 leader_election.cc:278] T bff7dab34ccd4b42842077e82106f6fb P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 23f962e4a1744381ad5fa0d2d8b10241: Illegal state: Tablet not RUNNING: NOT_STARTED W1031 23:06:38.965961 5815 leader_election.cc:278] T bff7dab34ccd4b42842077e82106f6fb P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 193181be29284287b25df2ee92938f0b: Not found: Tablet not found I1031 23:06:38.965989 5815 leader_election.cc:243] T bff7dab34ccd4b42842077e82106f6fb P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. I1031 23:06:38.966238 35627 raft_consensus.cc:2162] T bff7dab34ccd4b42842077e82106f6fb P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 6.660s I1031 23:06:38.966255 35627 raft_consensus.cc:1908] T bff7dab34ccd4b42842077e82106f6fb P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority W1031 23:06:39.090903 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 272 times. W1031 23:06:39.106844 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 332 times. I1031 23:06:39.548763 34590 raft_consensus.cc:411] T 117cd74bcdab4942901acb0005cdd702 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I1031 23:06:39.548813 34590 raft_consensus.cc:2162] T 117cd74bcdab4942901acb0005cdd702 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.598s I1031 23:06:39.548877 34590 raft_consensus.cc:435] T 117cd74bcdab4942901acb0005cdd702 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625f49b9554549e6a5c3a92b76e576cf" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st10.bj" port: 18700 } } peers { permanent_uuid: "0a8bbcbbb5a6459aa5258464755db3bb" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st02.bj" port: 18700 } } peers { permanent_uuid: "0f53a0d3ef7e44ebb0365c800752d5bd" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st01.bj" port: 18700 } } I1031 23:06:39.549615 34590 leader_election.cc:216] T 117cd74bcdab4942901acb0005cdd702 P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 625f49b9554549e6a5c3a92b76e576cf I1031 23:06:39.549643 34590 leader_election.cc:216] T 117cd74bcdab4942901acb0005cdd702 P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 0a8bbcbbb5a6459aa5258464755db3bb W1031 23:06:39.549837 5816 leader_election.cc:278] T 117cd74bcdab4942901acb0005cdd702 P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0a8bbcbbb5a6459aa5258464755db3bb: Not found: Tablet not found W1031 23:06:39.549934 5814 leader_election.cc:278] T 117cd74bcdab4942901acb0005cdd702 P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 625f49b9554549e6a5c3a92b76e576cf: Not found: Tablet not found I1031 23:06:39.549955 5814 leader_election.cc:243] T 117cd74bcdab4942901acb0005cdd702 P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. I1031 23:06:39.550230 35629 raft_consensus.cc:2162] T 117cd74bcdab4942901acb0005cdd702 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 11.871s I1031 23:06:39.550261 35629 raft_consensus.cc:1908] T 117cd74bcdab4942901acb0005cdd702 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority W1031 23:06:39.590981 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 273 times. W1031 23:06:39.606863 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 333 times. W1031 23:06:40.091085 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 274 times. W1031 23:06:40.107033 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 334 times. W1031 23:06:40.591120 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 275 times. W1031 23:06:40.607013 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 335 times. W1031 23:06:41.091140 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 276 times. W1031 23:06:41.107156 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 336 times. W1031 23:06:41.591259 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 277 times. W1031 23:06:41.607205 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 337 times. W1031 23:06:42.091416 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 278 times. W1031 23:06:42.107321 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 338 times. W1031 23:06:42.591454 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 279 times. W1031 23:06:42.607417 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 339 times. I1031 23:06:42.747928 34585 raft_consensus.cc:411] T 56155a2adc24450d89e338cc955bfa18 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I1031 23:06:42.747965 34585 raft_consensus.cc:2162] T 56155a2adc24450d89e338cc955bfa18 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 9.740s I1031 23:06:42.748019 34585 raft_consensus.cc:435] T 56155a2adc24450d89e338cc955bfa18 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625f49b9554549e6a5c3a92b76e576cf" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st10.bj" port: 18700 } } peers { permanent_uuid: "193181be29284287b25df2ee92938f0b" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st08.bj" port: 18700 } } peers { permanent_uuid: "0f53a0d3ef7e44ebb0365c800752d5bd" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st01.bj" port: 18700 } } I1031 23:06:42.748538 34585 leader_election.cc:216] T 56155a2adc24450d89e338cc955bfa18 P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 625f49b9554549e6a5c3a92b76e576cf I1031 23:06:42.748564 34585 leader_election.cc:216] T 56155a2adc24450d89e338cc955bfa18 P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 193181be29284287b25df2ee92938f0b W1031 23:06:42.748883 5815 leader_election.cc:278] T 56155a2adc24450d89e338cc955bfa18 P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 193181be29284287b25df2ee92938f0b: Not found: Tablet not found W1031 23:06:42.748942 5814 leader_election.cc:278] T 56155a2adc24450d89e338cc955bfa18 P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 625f49b9554549e6a5c3a92b76e576cf: Not found: Tablet not found I1031 23:06:42.748970 5814 leader_election.cc:243] T 56155a2adc24450d89e338cc955bfa18 P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. I1031 23:06:42.749714 35635 raft_consensus.cc:2162] T 56155a2adc24450d89e338cc955bfa18 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 8.088s I1031 23:06:42.749747 35635 raft_consensus.cc:1908] T 56155a2adc24450d89e338cc955bfa18 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority W1031 23:06:43.091574 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 280 times. W1031 23:06:43.107537 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 340 times. W1031 23:06:43.591660 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 281 times. W1031 23:06:43.607537 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 341 times. W1031 23:06:44.091789 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 282 times. W1031 23:06:44.107655 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 342 times. W1031 23:06:44.591935 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 283 times. W1031 23:06:44.607709 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 343 times. W1031 23:06:45.091923 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 284 times. W1031 23:06:45.107862 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 344 times. W1031 23:06:45.591991 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 285 times. W1031 23:06:45.607890 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 345 times. W1031 23:06:46.092044 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 286 times. W1031 23:06:46.107931 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 346 times. W1031 23:06:46.592068 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 287 times. W1031 23:06:46.608021 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 347 times. W1031 23:06:47.092212 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 288 times. W1031 23:06:47.108119 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 348 times. I1031 23:06:47.214886 34574 raft_consensus.cc:411] T bff7dab34ccd4b42842077e82106f6fb P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I1031 23:06:47.214929 34574 raft_consensus.cc:2162] T bff7dab34ccd4b42842077e82106f6fb P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 10.706s I1031 23:06:47.214977 34574 raft_consensus.cc:435] T bff7dab34ccd4b42842077e82106f6fb P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f53a0d3ef7e44ebb0365c800752d5bd" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st01.bj" port: 18700 } } peers { permanent_uuid: "23f962e4a1744381ad5fa0d2d8b10241" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st07.bj" port: 18700 } } peers { permanent_uuid: "193181be29284287b25df2ee92938f0b" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st08.bj" port: 18700 } } I1031 23:06:47.215543 34574 leader_election.cc:216] T bff7dab34ccd4b42842077e82106f6fb P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 23f962e4a1744381ad5fa0d2d8b10241 I1031 23:06:47.215569 34574 leader_election.cc:216] T bff7dab34ccd4b42842077e82106f6fb P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 193181be29284287b25df2ee92938f0b W1031 23:06:47.215778 5816 leader_election.cc:278] T bff7dab34ccd4b42842077e82106f6fb P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 23f962e4a1744381ad5fa0d2d8b10241: Illegal state: Tablet not RUNNING: NOT_STARTED W1031 23:06:47.215950 5815 leader_election.cc:278] T bff7dab34ccd4b42842077e82106f6fb P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 193181be29284287b25df2ee92938f0b: Not found: Tablet not found I1031 23:06:47.215989 5815 leader_election.cc:243] T bff7dab34ccd4b42842077e82106f6fb P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. I1031 23:06:47.216229 35654 raft_consensus.cc:2162] T bff7dab34ccd4b42842077e82106f6fb P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 19.172s I1031 23:06:47.216246 35654 raft_consensus.cc:1908] T bff7dab34ccd4b42842077e82106f6fb P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority W1031 23:06:47.592259 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 289 times. W1031 23:06:47.608111 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 349 times. W1031 23:06:48.092427 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 290 times. W1031 23:06:48.108332 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 350 times. W1031 23:06:48.592455 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 291 times. W1031 23:06:48.608361 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 351 times. W1031 23:06:49.092519 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 292 times. W1031 23:06:49.108443 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 352 times. W1031 23:06:49.592586 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 293 times. W1031 23:06:49.608568 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 353 times. W1031 23:06:50.092733 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 294 times. W1031 23:06:50.108603 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 354 times. W1031 23:06:50.592716 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 295 times. W1031 23:06:50.608677 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 355 times. W1031 23:06:51.092844 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 296 times. W1031 23:06:51.108803 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 356 times. W1031 23:06:51.592874 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 297 times. W1031 23:06:51.608899 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 357 times. W1031 23:06:52.093062 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 298 times. W1031 23:06:52.108976 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 358 times. W1031 23:06:52.593096 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 299 times. W1031 23:06:52.609089 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 359 times. I1031 23:06:52.981122 34585 raft_consensus.cc:411] T 56155a2adc24450d89e338cc955bfa18 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I1031 23:06:52.981184 34585 raft_consensus.cc:2162] T 56155a2adc24450d89e338cc955bfa18 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 7.791s I1031 23:06:52.981240 34585 raft_consensus.cc:435] T 56155a2adc24450d89e338cc955bfa18 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625f49b9554549e6a5c3a92b76e576cf" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st10.bj" port: 18700 } } peers { permanent_uuid: "193181be29284287b25df2ee92938f0b" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st08.bj" port: 18700 } } peers { permanent_uuid: "0f53a0d3ef7e44ebb0365c800752d5bd" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st01.bj" port: 18700 } } I1031 23:06:52.981849 34585 leader_election.cc:216] T 56155a2adc24450d89e338cc955bfa18 P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 625f49b9554549e6a5c3a92b76e576cf I1031 23:06:52.981886 34585 leader_election.cc:216] T 56155a2adc24450d89e338cc955bfa18 P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 193181be29284287b25df2ee92938f0b W1031 23:06:52.982182 5814 leader_election.cc:278] T 56155a2adc24450d89e338cc955bfa18 P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 625f49b9554549e6a5c3a92b76e576cf: Not found: Tablet not found W1031 23:06:52.982249 5815 leader_election.cc:278] T 56155a2adc24450d89e338cc955bfa18 P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 193181be29284287b25df2ee92938f0b: Not found: Tablet not found I1031 23:06:52.982275 5815 leader_election.cc:243] T 56155a2adc24450d89e338cc955bfa18 P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. I1031 23:06:52.982472 35666 raft_consensus.cc:2162] T 56155a2adc24450d89e338cc955bfa18 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 10.029s I1031 23:06:52.982504 35666 raft_consensus.cc:1908] T 56155a2adc24450d89e338cc955bfa18 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority I1031 23:06:53.015626 34590 raft_consensus.cc:411] T 117cd74bcdab4942901acb0005cdd702 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I1031 23:06:53.015660 34590 raft_consensus.cc:2162] T 117cd74bcdab4942901acb0005cdd702 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 14.581s I1031 23:06:53.015702 34590 raft_consensus.cc:435] T 117cd74bcdab4942901acb0005cdd702 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625f49b9554549e6a5c3a92b76e576cf" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st10.bj" port: 18700 } } peers { permanent_uuid: "0a8bbcbbb5a6459aa5258464755db3bb" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st02.bj" port: 18700 } } peers { permanent_uuid: "0f53a0d3ef7e44ebb0365c800752d5bd" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st01.bj" port: 18700 } } I1031 23:06:53.016353 34590 leader_election.cc:216] T 117cd74bcdab4942901acb0005cdd702 P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 625f49b9554549e6a5c3a92b76e576cf I1031 23:06:53.016388 34590 leader_election.cc:216] T 117cd74bcdab4942901acb0005cdd702 P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 0a8bbcbbb5a6459aa5258464755db3bb W1031 23:06:53.016618 5814 leader_election.cc:278] T 117cd74bcdab4942901acb0005cdd702 P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 625f49b9554549e6a5c3a92b76e576cf: Not found: Tablet not found W1031 23:06:53.016647 5816 leader_election.cc:278] T 117cd74bcdab4942901acb0005cdd702 P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0a8bbcbbb5a6459aa5258464755db3bb: Not found: Tablet not found I1031 23:06:53.016670 5816 leader_election.cc:243] T 117cd74bcdab4942901acb0005cdd702 P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. I1031 23:06:53.017549 35667 raft_consensus.cc:2162] T 117cd74bcdab4942901acb0005cdd702 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 9.962s I1031 23:06:53.017570 35667 raft_consensus.cc:1908] T 117cd74bcdab4942901acb0005cdd702 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority W1031 23:06:53.093287 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 300 times. W1031 23:06:53.109148 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 360 times. W1031 23:06:53.593307 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 301 times. W1031 23:06:53.609151 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 361 times. W1031 23:06:54.093369 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 302 times. W1031 23:06:54.109263 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 362 times. W1031 23:06:54.593447 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 303 times. W1031 23:06:54.609251 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 363 times. W1031 23:06:55.093488 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 304 times. W1031 23:06:55.109414 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 364 times. W1031 23:06:55.593530 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 305 times. W1031 23:06:55.609390 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 365 times. W1031 23:06:56.093667 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 306 times. W1031 23:06:56.109577 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 366 times. W1031 23:06:56.593709 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 307 times. W1031 23:06:56.609711 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 367 times. W1031 23:06:57.093838 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 308 times. W1031 23:06:57.109771 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 368 times. W1031 23:06:57.593896 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 309 times. W1031 23:06:57.609815 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 369 times. W1031 23:06:58.093950 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 310 times. W1031 23:06:58.109956 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 370 times. W1031 23:06:58.594072 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 311 times. W1031 23:06:58.609899 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 371 times. W1031 23:06:59.094095 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 312 times. W1031 23:06:59.110056 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 372 times. W1031 23:06:59.594142 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 313 times. W1031 23:06:59.610088 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 373 times. W1031 23:07:00.094297 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 314 times. W1031 23:07:00.110168 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 374 times. W1031 23:07:00.594322 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 315 times. W1031 23:07:00.610218 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 375 times. W1031 23:07:01.094557 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 316 times. W1031 23:07:01.110364 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 376 times. W1031 23:07:01.594615 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 317 times. W1031 23:07:01.610435 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 377 times. W1031 23:07:02.094647 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 318 times. W1031 23:07:02.110513 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 378 times. W1031 23:07:02.594671 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 319 times. W1031 23:07:02.610492 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 379 times. W1031 23:07:03.094771 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 320 times. W1031 23:07:03.110602 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 380 times. W1031 23:07:03.594879 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 321 times. W1031 23:07:03.610729 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 381 times. W1031 23:07:04.094988 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 322 times. W1031 23:07:04.110715 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 382 times. I1031 23:07:04.502136 34590 raft_consensus.cc:411] T 117cd74bcdab4942901acb0005cdd702 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I1031 23:07:04.502197 34590 raft_consensus.cc:2162] T 117cd74bcdab4942901acb0005cdd702 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 10.309s I1031 23:07:04.502256 34590 raft_consensus.cc:435] T 117cd74bcdab4942901acb0005cdd702 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625f49b9554549e6a5c3a92b76e576cf" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st10.bj" port: 18700 } } peers { permanent_uuid: "0a8bbcbbb5a6459aa5258464755db3bb" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st02.bj" port: 18700 } } peers { permanent_uuid: "0f53a0d3ef7e44ebb0365c800752d5bd" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st01.bj" port: 18700 } } I1031 23:07:04.502897 34590 leader_election.cc:216] T 117cd74bcdab4942901acb0005cdd702 P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 625f49b9554549e6a5c3a92b76e576cf I1031 23:07:04.502928 34590 leader_election.cc:216] T 117cd74bcdab4942901acb0005cdd702 P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 0a8bbcbbb5a6459aa5258464755db3bb W1031 23:07:04.503154 5814 leader_election.cc:278] T 117cd74bcdab4942901acb0005cdd702 P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 625f49b9554549e6a5c3a92b76e576cf: Not found: Tablet not found W1031 23:07:04.503247 5816 leader_election.cc:278] T 117cd74bcdab4942901acb0005cdd702 P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0a8bbcbbb5a6459aa5258464755db3bb: Not found: Tablet not found I1031 23:07:04.503268 5816 leader_election.cc:243] T 117cd74bcdab4942901acb0005cdd702 P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. I1031 23:07:04.504091 35704 raft_consensus.cc:2162] T 117cd74bcdab4942901acb0005cdd702 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 13.426s I1031 23:07:04.504113 35704 raft_consensus.cc:1908] T 117cd74bcdab4942901acb0005cdd702 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority W1031 23:07:04.594878 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 323 times. W1031 23:07:04.610844 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 383 times. W1031 23:07:05.095086 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 324 times. W1031 23:07:05.111086 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 384 times. I1031 23:07:05.167985 34585 raft_consensus.cc:411] T 56155a2adc24450d89e338cc955bfa18 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I1031 23:07:05.168025 34585 raft_consensus.cc:2162] T 56155a2adc24450d89e338cc955bfa18 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 12.279s I1031 23:07:05.168077 34585 raft_consensus.cc:435] T 56155a2adc24450d89e338cc955bfa18 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625f49b9554549e6a5c3a92b76e576cf" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st10.bj" port: 18700 } } peers { permanent_uuid: "193181be29284287b25df2ee92938f0b" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st08.bj" port: 18700 } } peers { permanent_uuid: "0f53a0d3ef7e44ebb0365c800752d5bd" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st01.bj" port: 18700 } } I1031 23:07:05.168584 34585 leader_election.cc:216] T 56155a2adc24450d89e338cc955bfa18 P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 625f49b9554549e6a5c3a92b76e576cf I1031 23:07:05.168620 34585 leader_election.cc:216] T 56155a2adc24450d89e338cc955bfa18 P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 193181be29284287b25df2ee92938f0b W1031 23:07:05.168882 5814 leader_election.cc:278] T 56155a2adc24450d89e338cc955bfa18 P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 625f49b9554549e6a5c3a92b76e576cf: Not found: Tablet not found W1031 23:07:05.168925 5815 leader_election.cc:278] T 56155a2adc24450d89e338cc955bfa18 P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 193181be29284287b25df2ee92938f0b: Not found: Tablet not found I1031 23:07:05.168956 5815 leader_election.cc:243] T 56155a2adc24450d89e338cc955bfa18 P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. I1031 23:07:05.169288 35722 raft_consensus.cc:2162] T 56155a2adc24450d89e338cc955bfa18 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 7.631s I1031 23:07:05.169311 35722 raft_consensus.cc:1908] T 56155a2adc24450d89e338cc955bfa18 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority W1031 23:07:05.595162 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 325 times. W1031 23:07:05.611003 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 385 times. W1031 23:07:06.095268 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 326 times. W1031 23:07:06.111083 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 386 times. W1031 23:07:06.595366 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 327 times. W1031 23:07:06.611155 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 387 times. W1031 23:07:07.095355 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 328 times. W1031 23:07:07.111194 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 388 times. W1031 23:07:07.595479 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 329 times. W1031 23:07:07.611250 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 389 times. W1031 23:07:08.095603 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 330 times. W1031 23:07:08.111271 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 390 times. I1031 23:07:08.160064 34574 raft_consensus.cc:411] T bff7dab34ccd4b42842077e82106f6fb P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I1031 23:07:08.160102 34574 raft_consensus.cc:2162] T bff7dab34ccd4b42842077e82106f6fb P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 12.824s I1031 23:07:08.160168 34574 raft_consensus.cc:435] T bff7dab34ccd4b42842077e82106f6fb P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f53a0d3ef7e44ebb0365c800752d5bd" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st01.bj" port: 18700 } } peers { permanent_uuid: "23f962e4a1744381ad5fa0d2d8b10241" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st07.bj" port: 18700 } } peers { permanent_uuid: "193181be29284287b25df2ee92938f0b" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st08.bj" port: 18700 } } I1031 23:07:08.160842 34574 leader_election.cc:216] T bff7dab34ccd4b42842077e82106f6fb P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 23f962e4a1744381ad5fa0d2d8b10241 I1031 23:07:08.160871 34574 leader_election.cc:216] T bff7dab34ccd4b42842077e82106f6fb P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 193181be29284287b25df2ee92938f0b W1031 23:07:08.161182 5816 leader_election.cc:278] T bff7dab34ccd4b42842077e82106f6fb P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 23f962e4a1744381ad5fa0d2d8b10241: Illegal state: Tablet not RUNNING: NOT_STARTED W1031 23:07:08.161226 5815 leader_election.cc:278] T bff7dab34ccd4b42842077e82106f6fb P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 193181be29284287b25df2ee92938f0b: Not found: Tablet not found I1031 23:07:08.161263 5815 leader_election.cc:243] T bff7dab34ccd4b42842077e82106f6fb P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. I1031 23:07:08.161545 35727 raft_consensus.cc:2162] T bff7dab34ccd4b42842077e82106f6fb P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 16.656s I1031 23:07:08.161567 35727 raft_consensus.cc:1908] T bff7dab34ccd4b42842077e82106f6fb P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority W1031 23:07:08.595582 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 331 times. W1031 23:07:08.611415 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 391 times. W1031 23:07:09.095759 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 332 times. W1031 23:07:09.111572 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 392 times. W1031 23:07:09.595700 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 333 times. W1031 23:07:09.611582 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 393 times. W1031 23:07:10.095852 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 334 times. W1031 23:07:10.111614 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 394 times. W1031 23:07:10.595939 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 335 times. W1031 23:07:10.611737 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 395 times. W1031 23:07:11.095939 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 336 times. W1031 23:07:11.111939 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 396 times. W1031 23:07:11.596068 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 337 times. W1031 23:07:11.612077 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 397 times. W1031 23:07:12.096155 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 338 times. W1031 23:07:12.112160 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 398 times. W1031 23:07:12.596160 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 339 times. W1031 23:07:12.612176 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 399 times. W1031 23:07:13.096364 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 340 times. W1031 23:07:13.112236 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 400 times. W1031 23:07:13.596429 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 341 times. W1031 23:07:13.612426 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 401 times. W1031 23:07:14.096500 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 342 times. W1031 23:07:14.112401 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 402 times. I1031 23:07:14.542562 34585 raft_consensus.cc:411] T 56155a2adc24450d89e338cc955bfa18 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I1031 23:07:14.542618 34585 raft_consensus.cc:2162] T 56155a2adc24450d89e338cc955bfa18 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 14.374s I1031 23:07:14.542670 34585 raft_consensus.cc:435] T 56155a2adc24450d89e338cc955bfa18 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625f49b9554549e6a5c3a92b76e576cf" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st10.bj" port: 18700 } } peers { permanent_uuid: "193181be29284287b25df2ee92938f0b" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st08.bj" port: 18700 } } peers { permanent_uuid: "0f53a0d3ef7e44ebb0365c800752d5bd" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st01.bj" port: 18700 } } I1031 23:07:14.543318 34585 leader_election.cc:216] T 56155a2adc24450d89e338cc955bfa18 P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 625f49b9554549e6a5c3a92b76e576cf I1031 23:07:14.543346 34585 leader_election.cc:216] T 56155a2adc24450d89e338cc955bfa18 P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 193181be29284287b25df2ee92938f0b W1031 23:07:14.543645 5814 leader_election.cc:278] T 56155a2adc24450d89e338cc955bfa18 P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 625f49b9554549e6a5c3a92b76e576cf: Not found: Tablet not found W1031 23:07:14.543696 5815 leader_election.cc:278] T 56155a2adc24450d89e338cc955bfa18 P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 193181be29284287b25df2ee92938f0b: Not found: Tablet not found I1031 23:07:14.543720 5815 leader_election.cc:243] T 56155a2adc24450d89e338cc955bfa18 P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. I1031 23:07:14.543988 35746 raft_consensus.cc:2162] T 56155a2adc24450d89e338cc955bfa18 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 15.390s I1031 23:07:14.544010 35746 raft_consensus.cc:1908] T 56155a2adc24450d89e338cc955bfa18 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority W1031 23:07:14.596534 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 343 times. W1031 23:07:14.612598 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 403 times. W1031 23:07:15.096678 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 344 times. W1031 23:07:15.112643 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 404 times. W1031 23:07:15.596741 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 345 times. W1031 23:07:15.612618 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 405 times. W1031 23:07:16.096875 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 346 times. W1031 23:07:16.112908 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 406 times. W1031 23:07:16.596906 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 347 times. W1031 23:07:16.612862 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 407 times. W1031 23:07:17.096899 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 348 times. W1031 23:07:17.112941 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 408 times. W1031 23:07:17.597077 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 349 times. W1031 23:07:17.613013 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 409 times. W1031 23:07:18.097194 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 350 times. W1031 23:07:18.113147 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 410 times. W1031 23:07:18.597208 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 351 times. W1031 23:07:18.613179 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 411 times. W1031 23:07:19.097368 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 352 times. W1031 23:07:19.113258 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 412 times. W1031 23:07:19.597451 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 353 times. W1031 23:07:19.613255 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 413 times. I1031 23:07:19.901837 34590 raft_consensus.cc:411] T 117cd74bcdab4942901acb0005cdd702 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I1031 23:07:19.901912 34590 raft_consensus.cc:2162] T 117cd74bcdab4942901acb0005cdd702 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 17.112s I1031 23:07:19.901993 34590 raft_consensus.cc:435] T 117cd74bcdab4942901acb0005cdd702 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625f49b9554549e6a5c3a92b76e576cf" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st10.bj" port: 18700 } } peers { permanent_uuid: "0a8bbcbbb5a6459aa5258464755db3bb" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st02.bj" port: 18700 } } peers { permanent_uuid: "0f53a0d3ef7e44ebb0365c800752d5bd" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st01.bj" port: 18700 } } I1031 23:07:19.902645 34590 leader_election.cc:216] T 117cd74bcdab4942901acb0005cdd702 P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 625f49b9554549e6a5c3a92b76e576cf I1031 23:07:19.902700 34590 leader_election.cc:216] T 117cd74bcdab4942901acb0005cdd702 P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 0a8bbcbbb5a6459aa5258464755db3bb W1031 23:07:19.903015 5814 leader_election.cc:278] T 117cd74bcdab4942901acb0005cdd702 P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 625f49b9554549e6a5c3a92b76e576cf: Not found: Tablet not found W1031 23:07:19.903084 5816 leader_election.cc:278] T 117cd74bcdab4942901acb0005cdd702 P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0a8bbcbbb5a6459aa5258464755db3bb: Not found: Tablet not found I1031 23:07:19.903110 5816 leader_election.cc:243] T 117cd74bcdab4942901acb0005cdd702 P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. I1031 23:07:19.903432 35994 raft_consensus.cc:2162] T 117cd74bcdab4942901acb0005cdd702 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 12.087s I1031 23:07:19.903450 35994 raft_consensus.cc:1908] T 117cd74bcdab4942901acb0005cdd702 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority W1031 23:07:20.097578 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 354 times. W1031 23:07:20.113459 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 414 times. W1031 23:07:20.597609 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 355 times. W1031 23:07:20.613564 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 415 times. W1031 23:07:21.097690 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 356 times. W1031 23:07:21.113656 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 416 times. W1031 23:07:21.597723 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 357 times. W1031 23:07:21.613698 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 417 times. W1031 23:07:22.097818 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 358 times. W1031 23:07:22.113814 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 418 times. W1031 23:07:22.597859 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 359 times. W1031 23:07:22.613962 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 419 times. W1031 23:07:23.098093 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 360 times. W1031 23:07:23.114017 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 420 times. W1031 23:07:23.598052 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 361 times. W1031 23:07:23.614128 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 421 times. W1031 23:07:24.098192 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 362 times. W1031 23:07:24.114181 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 422 times. W1031 23:07:24.598182 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 363 times. W1031 23:07:24.614279 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 423 times. W1031 23:07:25.098284 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 364 times. W1031 23:07:25.114395 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 424 times. W1031 23:07:25.598387 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 365 times. W1031 23:07:25.614495 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 425 times. W1031 23:07:26.098470 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 366 times. W1031 23:07:26.114573 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 426 times. I1031 23:07:26.395689 34574 raft_consensus.cc:411] T bff7dab34ccd4b42842077e82106f6fb P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I1031 23:07:26.395733 34574 raft_consensus.cc:2162] T bff7dab34ccd4b42842077e82106f6fb P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 8.887s I1031 23:07:26.395790 34574 raft_consensus.cc:435] T bff7dab34ccd4b42842077e82106f6fb P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f53a0d3ef7e44ebb0365c800752d5bd" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st01.bj" port: 18700 } } peers { permanent_uuid: "23f962e4a1744381ad5fa0d2d8b10241" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st07.bj" port: 18700 } } peers { permanent_uuid: "193181be29284287b25df2ee92938f0b" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st08.bj" port: 18700 } } I1031 23:07:26.396327 34574 leader_election.cc:216] T bff7dab34ccd4b42842077e82106f6fb P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 23f962e4a1744381ad5fa0d2d8b10241 I1031 23:07:26.396354 34574 leader_election.cc:216] T bff7dab34ccd4b42842077e82106f6fb P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 193181be29284287b25df2ee92938f0b W1031 23:07:26.396605 5815 leader_election.cc:278] T bff7dab34ccd4b42842077e82106f6fb P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 193181be29284287b25df2ee92938f0b: Not found: Tablet not found W1031 23:07:26.396658 5816 leader_election.cc:278] T bff7dab34ccd4b42842077e82106f6fb P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 23f962e4a1744381ad5fa0d2d8b10241: Illegal state: Tablet not RUNNING: NOT_STARTED I1031 23:07:26.396693 5816 leader_election.cc:243] T bff7dab34ccd4b42842077e82106f6fb P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. I1031 23:07:26.397027 36016 raft_consensus.cc:2162] T bff7dab34ccd4b42842077e82106f6fb P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.671s I1031 23:07:26.397059 36016 raft_consensus.cc:1908] T bff7dab34ccd4b42842077e82106f6fb P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority W1031 23:07:26.598536 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 367 times. W1031 23:07:26.614665 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 427 times. W1031 23:07:27.098584 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 368 times. W1031 23:07:27.114747 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 428 times. W1031 23:07:27.598701 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 369 times. W1031 23:07:27.614831 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 429 times. W1031 23:07:28.098776 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 370 times. W1031 23:07:28.114915 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 430 times. I1031 23:07:28.296468 5866 tablet_service.cc:691] Processing DeleteTablet for tablet 117cd74bcdab4942901acb0005cdd702 with delete_type TABLET_DATA_DELETED (Replaced by acb401e40e7049fbbfdbeb40474353d9 at 2017-10-31 23:04:52 CST) from {username='work'} at 10.108.187.10:53997 I1031 23:07:28.345407 5866 tablet_service.cc:691] Processing DeleteTablet for tablet bff7dab34ccd4b42842077e82106f6fb with delete_type TABLET_DATA_DELETED (Replaced by 005a48fb466c4a52896842dd00124755 at 2017-10-31 23:04:52 CST) from {username='work'} at 10.108.187.10:53997 I1031 23:07:28.384330 5866 tablet_service.cc:691] Processing DeleteTablet for tablet 56155a2adc24450d89e338cc955bfa18 with delete_type TABLET_DATA_DELETED (Replaced by 80ba7f33ec864c5f96c61932edb5cdc4 at 2017-10-31 23:04:52 CST) from {username='work'} at 10.108.187.10:53997 W1031 23:07:28.598850 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 371 times. W1031 23:07:28.614928 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 431 times. W1031 23:07:29.098939 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 372 times. W1031 23:07:29.115087 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 432 times. I1031 23:07:29.590828 34574 raft_consensus.cc:411] T bff7dab34ccd4b42842077e82106f6fb P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I1031 23:07:29.590873 34574 raft_consensus.cc:2162] T bff7dab34ccd4b42842077e82106f6fb P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 9.341s I1031 23:07:29.590934 34574 raft_consensus.cc:435] T bff7dab34ccd4b42842077e82106f6fb P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f53a0d3ef7e44ebb0365c800752d5bd" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st01.bj" port: 18700 } } peers { permanent_uuid: "23f962e4a1744381ad5fa0d2d8b10241" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st07.bj" port: 18700 } } peers { permanent_uuid: "193181be29284287b25df2ee92938f0b" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st08.bj" port: 18700 } } I1031 23:07:29.591612 34574 leader_election.cc:216] T bff7dab34ccd4b42842077e82106f6fb P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 23f962e4a1744381ad5fa0d2d8b10241 I1031 23:07:29.591646 34574 leader_election.cc:216] T bff7dab34ccd4b42842077e82106f6fb P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 193181be29284287b25df2ee92938f0b W1031 23:07:29.591934 5816 leader_election.cc:278] T bff7dab34ccd4b42842077e82106f6fb P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 23f962e4a1744381ad5fa0d2d8b10241: Illegal state: Tablet not RUNNING: NOT_STARTED W1031 23:07:29.591998 5815 leader_election.cc:278] T bff7dab34ccd4b42842077e82106f6fb P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 193181be29284287b25df2ee92938f0b: Not found: Tablet not found I1031 23:07:29.592054 5815 leader_election.cc:243] T bff7dab34ccd4b42842077e82106f6fb P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. I1031 23:07:29.592835 36022 raft_consensus.cc:2162] T bff7dab34ccd4b42842077e82106f6fb P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 5.159s I1031 23:07:29.592869 36022 raft_consensus.cc:1908] T bff7dab34ccd4b42842077e82106f6fb P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority W1031 23:07:29.598970 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 373 times. W1031 23:07:29.614977 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 433 times. W1031 23:07:30.099066 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 374 times. W1031 23:07:30.115303 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 434 times. W1031 23:07:30.599110 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 375 times. W1031 23:07:30.615262 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 435 times. W1031 23:07:31.099164 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 376 times. W1031 23:07:31.115456 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 436 times. W1031 23:07:31.599367 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 377 times. W1031 23:07:31.615602 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 437 times. I1031 23:07:31.713057 34585 raft_consensus.cc:411] T 56155a2adc24450d89e338cc955bfa18 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I1031 23:07:31.713096 34585 raft_consensus.cc:2162] T 56155a2adc24450d89e338cc955bfa18 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 13.708s I1031 23:07:31.713168 34585 raft_consensus.cc:435] T 56155a2adc24450d89e338cc955bfa18 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625f49b9554549e6a5c3a92b76e576cf" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st10.bj" port: 18700 } } peers { permanent_uuid: "193181be29284287b25df2ee92938f0b" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st08.bj" port: 18700 } } peers { permanent_uuid: "0f53a0d3ef7e44ebb0365c800752d5bd" member_type: VOTER last_known_addr { host: "c3-kudu-tst-st01.bj" port: 18700 } } I1031 23:07:31.713737 34585 leader_election.cc:216] T 56155a2adc24450d89e338cc955bfa18 P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 625f49b9554549e6a5c3a92b76e576cf I1031 23:07:31.713760 34585 leader_election.cc:216] T 56155a2adc24450d89e338cc955bfa18 P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 193181be29284287b25df2ee92938f0b W1031 23:07:31.714020 5815 leader_election.cc:278] T 56155a2adc24450d89e338cc955bfa18 P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 193181be29284287b25df2ee92938f0b: Not found: Tablet not found W1031 23:07:31.714100 5814 leader_election.cc:278] T 56155a2adc24450d89e338cc955bfa18 P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 625f49b9554549e6a5c3a92b76e576cf: Not found: Tablet not found I1031 23:07:31.714128 5814 leader_election.cc:243] T 56155a2adc24450d89e338cc955bfa18 P 0f53a0d3ef7e44ebb0365c800752d5bd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. I1031 23:07:31.714936 36027 raft_consensus.cc:2162] T 56155a2adc24450d89e338cc955bfa18 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 3.213s I1031 23:07:31.714968 36027 raft_consensus.cc:1908] T 56155a2adc24450d89e338cc955bfa18 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority W1031 23:07:32.099459 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 378 times. W1031 23:07:32.115670 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 438 times. I1031 23:07:32.373733 5853 maintenance_manager.cc:208] P 0f53a0d3ef7e44ebb0365c800752d5bd: Unregistered op CompactRowSetsOp(117cd74bcdab4942901acb0005cdd702) I1031 23:07:32.373769 5857 maintenance_manager.cc:208] P 0f53a0d3ef7e44ebb0365c800752d5bd: Unregistered op CompactRowSetsOp(bff7dab34ccd4b42842077e82106f6fb) I1031 23:07:32.373783 5861 maintenance_manager.cc:208] P 0f53a0d3ef7e44ebb0365c800752d5bd: Unregistered op CompactRowSetsOp(56155a2adc24450d89e338cc955bfa18) I1031 23:07:32.373793 5853 maintenance_manager.cc:208] P 0f53a0d3ef7e44ebb0365c800752d5bd: Unregistered op MinorDeltaCompactionOp(117cd74bcdab4942901acb0005cdd702) I1031 23:07:32.373843 5857 maintenance_manager.cc:208] P 0f53a0d3ef7e44ebb0365c800752d5bd: Unregistered op MinorDeltaCompactionOp(bff7dab34ccd4b42842077e82106f6fb) I1031 23:07:32.373852 5853 maintenance_manager.cc:208] P 0f53a0d3ef7e44ebb0365c800752d5bd: Unregistered op MajorDeltaCompactionOp(117cd74bcdab4942901acb0005cdd702) I1031 23:07:32.373867 5857 maintenance_manager.cc:208] P 0f53a0d3ef7e44ebb0365c800752d5bd: Unregistered op MajorDeltaCompactionOp(bff7dab34ccd4b42842077e82106f6fb) I1031 23:07:32.373878 5853 maintenance_manager.cc:208] P 0f53a0d3ef7e44ebb0365c800752d5bd: Unregistered op UndoDeltaBlockGCOp(117cd74bcdab4942901acb0005cdd702) I1031 23:07:32.373885 5857 maintenance_manager.cc:208] P 0f53a0d3ef7e44ebb0365c800752d5bd: Unregistered op UndoDeltaBlockGCOp(bff7dab34ccd4b42842077e82106f6fb) I1031 23:07:32.373900 5861 maintenance_manager.cc:208] P 0f53a0d3ef7e44ebb0365c800752d5bd: Unregistered op MinorDeltaCompactionOp(56155a2adc24450d89e338cc955bfa18) I1031 23:07:32.373915 5853 maintenance_manager.cc:208] P 0f53a0d3ef7e44ebb0365c800752d5bd: Unregistered op FlushMRSOp(117cd74bcdab4942901acb0005cdd702) I1031 23:07:32.373921 5861 maintenance_manager.cc:208] P 0f53a0d3ef7e44ebb0365c800752d5bd: Unregistered op MajorDeltaCompactionOp(56155a2adc24450d89e338cc955bfa18) I1031 23:07:32.373944 5853 maintenance_manager.cc:208] P 0f53a0d3ef7e44ebb0365c800752d5bd: Unregistered op FlushDeltaMemStoresOp(117cd74bcdab4942901acb0005cdd702) I1031 23:07:32.373951 5861 maintenance_manager.cc:208] P 0f53a0d3ef7e44ebb0365c800752d5bd: Unregistered op UndoDeltaBlockGCOp(56155a2adc24450d89e338cc955bfa18) I1031 23:07:32.373965 5857 maintenance_manager.cc:208] P 0f53a0d3ef7e44ebb0365c800752d5bd: Unregistered op FlushMRSOp(bff7dab34ccd4b42842077e82106f6fb) I1031 23:07:32.373975 5861 maintenance_manager.cc:208] P 0f53a0d3ef7e44ebb0365c800752d5bd: Unregistered op FlushMRSOp(56155a2adc24450d89e338cc955bfa18) I1031 23:07:32.373982 5853 maintenance_manager.cc:208] P 0f53a0d3ef7e44ebb0365c800752d5bd: Unregistered op LogGCOp(117cd74bcdab4942901acb0005cdd702) I1031 23:07:32.373991 5857 maintenance_manager.cc:208] P 0f53a0d3ef7e44ebb0365c800752d5bd: Unregistered op FlushDeltaMemStoresOp(bff7dab34ccd4b42842077e82106f6fb) I1031 23:07:32.373999 5853 raft_consensus.cc:1569] T 117cd74bcdab4942901acb0005cdd702 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Raft consensus shutting down. I1031 23:07:32.374012 5861 maintenance_manager.cc:208] P 0f53a0d3ef7e44ebb0365c800752d5bd: Unregistered op FlushDeltaMemStoresOp(56155a2adc24450d89e338cc955bfa18) I1031 23:07:32.374018 5853 raft_consensus.cc:1585] T 117cd74bcdab4942901acb0005cdd702 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Raft consensus is shut down! I1031 23:07:32.374020 5857 maintenance_manager.cc:208] P 0f53a0d3ef7e44ebb0365c800752d5bd: Unregistered op LogGCOp(bff7dab34ccd4b42842077e82106f6fb) I1031 23:07:32.374115 5857 raft_consensus.cc:1569] T bff7dab34ccd4b42842077e82106f6fb P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Raft consensus shutting down. I1031 23:07:32.374136 5857 raft_consensus.cc:1585] T bff7dab34ccd4b42842077e82106f6fb P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Raft consensus is shut down! I1031 23:07:32.374030 5861 maintenance_manager.cc:208] P 0f53a0d3ef7e44ebb0365c800752d5bd: Unregistered op LogGCOp(56155a2adc24450d89e338cc955bfa18) I1031 23:07:32.374207 5861 raft_consensus.cc:1569] T 56155a2adc24450d89e338cc955bfa18 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Raft consensus shutting down. I1031 23:07:32.374254 5861 raft_consensus.cc:1585] T 56155a2adc24450d89e338cc955bfa18 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 0 FOLLOWER]: Raft consensus is shut down! I1031 23:07:32.377606 5857 ts_tablet_manager.cc:1020] T bff7dab34ccd4b42842077e82106f6fb P 0f53a0d3ef7e44ebb0365c800752d5bd: Deleting tablet data with delete state TABLET_DATA_DELETED I1031 23:07:32.379412 5853 ts_tablet_manager.cc:1020] T 117cd74bcdab4942901acb0005cdd702 P 0f53a0d3ef7e44ebb0365c800752d5bd: Deleting tablet data with delete state TABLET_DATA_DELETED I1031 23:07:32.379539 5861 ts_tablet_manager.cc:1020] T 56155a2adc24450d89e338cc955bfa18 P 0f53a0d3ef7e44ebb0365c800752d5bd: Deleting tablet data with delete state TABLET_DATA_DELETED I1031 23:07:32.548648 5853 ts_tablet_manager.cc:1032] T 117cd74bcdab4942901acb0005cdd702 P 0f53a0d3ef7e44ebb0365c800752d5bd: Tablet deleted. Last logged OpId: 0.0 I1031 23:07:32.548774 5853 log.cc:876] T 117cd74bcdab4942901acb0005cdd702 P 0f53a0d3ef7e44ebb0365c800752d5bd: Deleting WAL directory at /home/work/app/kudu/c3prc-kernel65/kudu_tablet_server/hdd1/wals/117cd74bcdab4942901acb0005cdd702 I1031 23:07:32.549101 5853 consensus_meta.cc:71] T 117cd74bcdab4942901acb0005cdd702 Deleting consensus metadata I1031 23:07:32.582350 5861 ts_tablet_manager.cc:1032] T 56155a2adc24450d89e338cc955bfa18 P 0f53a0d3ef7e44ebb0365c800752d5bd: Tablet deleted. Last logged OpId: 0.0 I1031 23:07:32.582494 5861 log.cc:876] T 56155a2adc24450d89e338cc955bfa18 P 0f53a0d3ef7e44ebb0365c800752d5bd: Deleting WAL directory at /home/work/app/kudu/c3prc-kernel65/kudu_tablet_server/hdd1/wals/56155a2adc24450d89e338cc955bfa18 I1031 23:07:32.582861 5861 consensus_meta.cc:71] T 56155a2adc24450d89e338cc955bfa18 Deleting consensus metadata W1031 23:07:32.599458 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 379 times. I1031 23:07:32.607692 5857 ts_tablet_manager.cc:1032] T bff7dab34ccd4b42842077e82106f6fb P 0f53a0d3ef7e44ebb0365c800752d5bd: Tablet deleted. Last logged OpId: 0.0 W1031 23:07:32.607830 5853 rpcz_store.cc:234] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 10.108.187.10:53997 (request call id 64) took 94334ms (client timeout 29999). W1031 23:07:32.607830 5861 rpcz_store.cc:234] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 10.108.187.10:53997 (request call id 67) took 94229ms (client timeout 29999). I1031 23:07:32.607867 5857 log.cc:876] T bff7dab34ccd4b42842077e82106f6fb P 0f53a0d3ef7e44ebb0365c800752d5bd: Deleting WAL directory at /home/work/app/kudu/c3prc-kernel65/kudu_tablet_server/hdd1/wals/bff7dab34ccd4b42842077e82106f6fb W1031 23:07:32.608037 5853 rpcz_store.cc:238] Trace: 1031 23:05:58.273724 (+ 0us) service_pool.cc:143] Inserting onto call queue 1031 23:05:58.273793 (+ 69us) service_pool.cc:202] Handling call 1031 23:05:58.273835 (+ 42us) ts_tablet_manager.cc:572] Deleting tablet 117cd74bcdab4942901acb0005cdd702 1031 23:05:58.273835 (+ 0us) ts_tablet_manager.cc:580] Acquired tablet manager lock 1031 23:07:32.464732 (+94190897us) tablet_metadata.cc:461] Metadata flushed 1031 23:07:32.548626 (+ 83894us) tablet_metadata.cc:461] Metadata flushed 1031 23:07:32.607816 (+ 59190us) inbound_call.cc:130] Queueing success response Metrics: {"fdatasync":2,"fdatasync_us":83904,"mutex_wait_us":94099890,"tcmalloc_contention_cycles":133376} W1031 23:07:32.608047 5861 rpcz_store.cc:238] Trace: 1031 23:05:58.377963 (+ 0us) service_pool.cc:143] Inserting onto call queue 1031 23:05:58.378009 (+ 46us) service_pool.cc:202] Handling call 1031 23:05:58.378109 (+ 100us) ts_tablet_manager.cc:572] Deleting tablet 56155a2adc24450d89e338cc955bfa18 1031 23:05:58.378110 (+ 1us) ts_tablet_manager.cc:580] Acquired tablet manager lock 1031 23:07:32.515015 (+94136905us) tablet_metadata.cc:461] Metadata flushed 1031 23:07:32.582317 (+ 67302us) tablet_metadata.cc:461] Metadata flushed 1031 23:07:32.607809 (+ 25492us) inbound_call.cc:130] Queueing success response Metrics: {"fdatasync":2,"fdatasync_us":83784,"mutex_wait_us":93995649,"tcmalloc_contention_cycles":76160} I1031 23:07:32.608186 5857 consensus_meta.cc:71] T bff7dab34ccd4b42842077e82106f6fb Deleting consensus metadata W1031 23:07:32.608386 5857 rpcz_store.cc:234] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 10.108.187.10:53997 (request call id 66) took 94277ms (client timeout 29999). W1031 23:07:32.608525 5857 rpcz_store.cc:238] Trace: 1031 23:05:58.330776 (+ 0us) service_pool.cc:143] Inserting onto call queue 1031 23:05:58.330787 (+ 11us) service_pool.cc:202] Handling call 1031 23:05:58.330824 (+ 37us) ts_tablet_manager.cc:572] Deleting tablet bff7dab34ccd4b42842077e82106f6fb 1031 23:05:58.330825 (+ 1us) ts_tablet_manager.cc:580] Acquired tablet manager lock 1031 23:07:32.498352 (+94167527us) tablet_metadata.cc:461] Metadata flushed 1031 23:07:32.607652 (+109300us) tablet_metadata.cc:461] Metadata flushed 1031 23:07:32.608374 (+ 722us) inbound_call.cc:130] Queueing success response Metrics: {"fdatasync":2,"fdatasync_us":85586,"mutex_wait_us":94042959,"tcmalloc_contention_cycles":43904} I1031 23:07:32.608857 5734 tablet.cc:911] T cd5d028ce30a4eb19eacbecbf5a3486b P 0f53a0d3ef7e44ebb0365c800752d5bd: Flush: entering stage 1 (old memrowset already frozen for inserts) I1031 23:07:32.608882 5734 compaction.cc:877] Selected 1 rowsets to compact: I1031 23:07:32.608907 5734 compaction.cc:880] memrowset(current size on disk: ~0 bytes) I1031 23:07:32.608922 5734 tablet.cc:913] T cd5d028ce30a4eb19eacbecbf5a3486b P 0f53a0d3ef7e44ebb0365c800752d5bd: Memstore in-memory size: 3751 bytes I1031 23:07:32.608947 5734 tablet.cc:1174] T cd5d028ce30a4eb19eacbecbf5a3486b P 0f53a0d3ef7e44ebb0365c800752d5bd: Flush: entering phase 1 (flushing snapshot). Phase 1 snapshot: MvccSnapshot[committed={T|T < 6182757230424834048 or (T in {6182757230424834048})}] I1031 23:07:32.609093 5734 multi_column_writer.cc:85] Opened CFile writer for column ts[int32 NOT NULL] I1031 23:07:32.609134 5734 multi_column_writer.cc:85] Opened CFile writer for column id[int32 NOT NULL] W1031 23:07:32.615627 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 439 times. I1031 23:07:32.739135 5734 tablet.cc:1259] T cd5d028ce30a4eb19eacbecbf5a3486b P 0f53a0d3ef7e44ebb0365c800752d5bd: Flush: entering phase 2 (starting to duplicate updates in new rowsets) I1031 23:07:32.967370 5734 tablet.cc:1314] T cd5d028ce30a4eb19eacbecbf5a3486b P 0f53a0d3ef7e44ebb0365c800752d5bd: Flush Phase 2: carrying over any updates which arrived during Phase 1 I1031 23:07:32.967432 5734 tablet.cc:1316] T cd5d028ce30a4eb19eacbecbf5a3486b P 0f53a0d3ef7e44ebb0365c800752d5bd: Phase 2 snapshot: MvccSnapshot[committed={T|T < 6182757230424834048 or (T in {6182757230424834048})}] W1031 23:07:33.099637 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 380 times. W1031 23:07:33.115788 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 440 times. I1031 23:07:33.390091 5734 tablet.cc:1358] T cd5d028ce30a4eb19eacbecbf5a3486b P 0f53a0d3ef7e44ebb0365c800752d5bd: Flush successful on 9 rows (4328 bytes) I1031 23:07:33.390180 5734 maintenance_manager.cc:419] Time spent running FlushMRSOp(cd5d028ce30a4eb19eacbecbf5a3486b): real 1.016s user 0.816s sys 0.004s I1031 23:07:33.390348 5734 maintenance_manager.cc:425] P 0f53a0d3ef7e44ebb0365c800752d5bd: FlushMRSOp(cd5d028ce30a4eb19eacbecbf5a3486b) metrics: {"cfile_init":1,"fdatasync":11,"fdatasync_us":173593,"lbm_read_time_us":7,"lbm_reads_lt_1ms":4,"lbm_write_time_us":175,"lbm_writes_lt_1ms":26,"tcmalloc_contention_cycles":98560} W1031 23:07:33.599798 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 381 times. W1031 23:07:33.615921 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 441 times. W1031 23:07:34.099807 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 382 times. W1031 23:07:34.115983 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 442 times. W1031 23:07:34.599928 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 383 times. W1031 23:07:34.616008 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 443 times. W1031 23:07:35.099884 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 384 times. W1031 23:07:35.116098 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 444 times. W1031 23:07:35.599997 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 385 times. W1031 23:07:35.616158 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 445 times. W1031 23:07:36.100064 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 386 times. W1031 23:07:36.116215 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 446 times. W1031 23:07:36.600114 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 387 times. W1031 23:07:36.616263 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 447 times. W1031 23:07:37.100260 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 388 times. W1031 23:07:37.116371 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 448 times. W1031 23:07:37.600304 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 389 times. W1031 23:07:37.616420 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 449 times. W1031 23:07:38.100414 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 390 times. W1031 23:07:38.116483 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 450 times. W1031 23:07:38.600469 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 391 times. W1031 23:07:38.616503 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 451 times. W1031 23:07:39.100576 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 392 times. W1031 23:07:39.116612 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 452 times. W1031 23:07:39.600700 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 393 times. W1031 23:07:39.616761 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 453 times. W1031 23:07:40.100800 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 394 times. W1031 23:07:40.116852 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 454 times. W1031 23:07:40.600886 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 395 times. W1031 23:07:40.616891 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 455 times. W1031 23:07:41.101006 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 396 times. W1031 23:07:41.117010 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 456 times. W1031 23:07:41.600983 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 397 times. W1031 23:07:41.617091 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 457 times. W1031 23:07:42.101119 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 398 times. W1031 23:07:42.117167 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 458 times. W1031 23:07:42.601275 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 399 times. W1031 23:07:42.617251 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 459 times. W1031 23:07:43.101361 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 400 times. W1031 23:07:43.117303 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 460 times. W1031 23:07:43.601411 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 401 times. W1031 23:07:43.617396 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 461 times. W1031 23:07:44.101455 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 402 times. W1031 23:07:44.117527 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 462 times. W1031 23:07:44.601536 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 403 times. W1031 23:07:44.617564 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 463 times. W1031 23:07:45.101626 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 404 times. W1031 23:07:45.117697 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 464 times. W1031 23:07:45.601750 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 405 times. W1031 23:07:45.617748 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 465 times. W1031 23:07:46.101778 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 406 times. W1031 23:07:46.117828 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 466 times. W1031 23:07:46.601896 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 407 times. W1031 23:07:46.617913 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 467 times. W1031 23:07:47.101974 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 408 times. W1031 23:07:47.117972 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 468 times. W1031 23:07:47.602088 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 409 times. W1031 23:07:47.618080 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 469 times. W1031 23:07:48.102156 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 410 times. W1031 23:07:48.118129 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 470 times. W1031 23:07:48.602224 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 411 times. W1031 23:07:48.618187 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 471 times. W1031 23:07:49.102238 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 412 times. W1031 23:07:49.118261 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 472 times. W1031 23:07:49.602427 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 413 times. W1031 23:07:49.618330 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 473 times. W1031 23:07:50.102491 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 414 times. W1031 23:07:50.118422 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 474 times. W1031 23:07:50.602640 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 415 times. W1031 23:07:50.618520 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 475 times. W1031 23:07:51.102721 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 416 times. W1031 23:07:51.118674 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 476 times. W1031 23:07:51.602731 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 417 times. W1031 23:07:51.618657 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 477 times. W1031 23:07:52.102824 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 418 times. W1031 23:07:52.118783 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 478 times. W1031 23:07:52.602959 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 419 times. W1031 23:07:52.618870 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 479 times. W1031 23:07:53.102972 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 420 times. W1031 23:07:53.118924 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 480 times. W1031 23:07:53.603085 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 421 times. W1031 23:07:53.619001 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 481 times. W1031 23:07:54.103107 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 422 times. W1031 23:07:54.119045 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 482 times. W1031 23:07:54.603184 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 423 times. W1031 23:07:54.619160 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 483 times. W1031 23:07:55.103327 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 424 times. W1031 23:07:55.119284 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 484 times. W1031 23:07:55.603467 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 425 times. W1031 23:07:55.619396 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 485 times. W1031 23:07:56.103497 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 426 times. W1031 23:07:56.119449 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 486 times. W1031 23:07:56.603533 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 427 times. W1031 23:07:56.619509 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 487 times. W1031 23:07:57.103662 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 428 times. W1031 23:07:57.119601 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 488 times. W1031 23:07:57.603680 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 429 times. W1031 23:07:57.619657 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 489 times. W1031 23:07:58.103837 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 430 times. W1031 23:07:58.119779 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 490 times. W1031 23:07:58.603948 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 431 times. W1031 23:07:58.619804 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 491 times. W1031 23:07:59.103966 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 432 times. W1031 23:07:59.120013 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 492 times. W1031 23:07:59.604022 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 433 times. W1031 23:07:59.620007 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 493 times. W1031 23:08:00.104105 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 434 times. W1031 23:08:00.120091 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 494 times. W1031 23:08:00.604216 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 435 times. W1031 23:08:00.620160 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 495 times. W1031 23:08:01.104279 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 436 times. W1031 23:08:01.120254 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 496 times. W1031 23:08:01.604387 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 437 times. W1031 23:08:01.620306 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 497 times. W1031 23:08:02.104482 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 438 times. W1031 23:08:02.120488 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 498 times. W1031 23:08:02.604562 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 439 times. W1031 23:08:02.620494 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 499 times. W1031 23:08:03.104676 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 440 times. W1031 23:08:03.120616 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 500 times. W1031 23:08:03.604811 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 441 times. W1031 23:08:03.620702 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 501 times. W1031 23:08:04.104812 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 442 times. W1031 23:08:04.120793 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 502 times. W1031 23:08:04.604794 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 443 times. W1031 23:08:04.620867 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 503 times. W1031 23:08:05.104902 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 444 times. W1031 23:08:05.120954 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 504 times. W1031 23:08:05.605072 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 445 times. W1031 23:08:05.620992 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 505 times. W1031 23:08:06.105090 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 446 times. W1031 23:08:06.120998 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 506 times. W1031 23:08:06.605237 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 447 times. W1031 23:08:06.621162 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 507 times. W1031 23:08:07.105231 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 448 times. W1031 23:08:07.121230 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 508 times. W1031 23:08:07.605324 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 449 times. W1031 23:08:07.621325 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 509 times. W1031 23:08:08.105415 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 450 times. W1031 23:08:08.121388 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 510 times. W1031 23:08:08.605551 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 451 times. W1031 23:08:08.621491 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 511 times. W1031 23:08:09.105512 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 452 times. W1031 23:08:09.121572 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 512 times. W1031 23:08:09.605720 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 453 times. W1031 23:08:09.621574 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 513 times. W1031 23:08:10.105813 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 454 times. W1031 23:08:10.121706 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 514 times. W1031 23:08:10.605808 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 455 times. W1031 23:08:10.621681 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 515 times. W1031 23:08:11.105832 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 456 times. W1031 23:08:11.121807 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 516 times. W1031 23:08:11.606005 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 457 times. W1031 23:08:11.621927 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 517 times. W1031 23:08:12.106048 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 458 times. W1031 23:08:12.121986 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 518 times. W1031 23:08:12.606168 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 459 times. W1031 23:08:12.622040 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 519 times. W1031 23:08:13.106240 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 460 times. W1031 23:08:13.122120 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 520 times. W1031 23:08:13.606256 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 461 times. W1031 23:08:13.622197 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 521 times. W1031 23:08:14.106490 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 462 times. W1031 23:08:14.122352 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 522 times. W1031 23:08:14.606472 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 463 times. W1031 23:08:14.622382 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 523 times. W1031 23:08:15.106644 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 464 times. W1031 23:08:15.122562 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 524 times. W1031 23:08:15.606564 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 465 times. W1031 23:08:15.622673 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 525 times. W1031 23:08:16.106766 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 466 times. W1031 23:08:16.122625 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 526 times. W1031 23:08:16.606891 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 467 times. W1031 23:08:16.622879 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 527 times. W1031 23:08:17.106899 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 468 times. W1031 23:08:17.122812 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 528 times. W1031 23:08:17.606983 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 469 times. W1031 23:08:17.622965 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 529 times. W1031 23:08:18.107108 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 470 times. W1031 23:08:18.123008 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 530 times. W1031 23:08:18.607172 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 471 times. W1031 23:08:18.623131 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 531 times. W1031 23:08:19.107190 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 472 times. W1031 23:08:19.123163 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 532 times. W1031 23:08:19.607249 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 473 times. W1031 23:08:19.623306 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 533 times. W1031 23:08:20.107376 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 474 times. W1031 23:08:20.123426 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 534 times. W1031 23:08:20.607481 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 475 times. W1031 23:08:20.623489 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 535 times. W1031 23:08:21.107576 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 476 times. W1031 23:08:21.123646 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 536 times. W1031 23:08:21.607625 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 477 times. W1031 23:08:21.623751 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 537 times. W1031 23:08:22.107700 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 478 times. W1031 23:08:22.123713 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 538 times. W1031 23:08:22.607784 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 479 times. W1031 23:08:22.623780 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 539 times. W1031 23:08:23.107861 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 480 times. W1031 23:08:23.123955 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 540 times. W1031 23:08:23.607964 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 481 times. W1031 23:08:23.623900 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 541 times. W1031 23:08:24.108062 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 482 times. W1031 23:08:24.124008 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 542 times. W1031 23:08:24.608101 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 483 times. W1031 23:08:24.624159 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 543 times. W1031 23:08:25.108247 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 484 times. W1031 23:08:25.124286 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 544 times. W1031 23:08:25.608322 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 485 times. W1031 23:08:25.624341 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 545 times. W1031 23:08:26.108494 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 486 times. W1031 23:08:26.124418 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 546 times. W1031 23:08:26.608486 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 487 times. W1031 23:08:26.624519 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 547 times. W1031 23:08:27.108587 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 488 times. W1031 23:08:27.124655 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 548 times. W1031 23:08:27.608665 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 489 times. W1031 23:08:27.624681 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 549 times. W1031 23:08:28.108824 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 490 times. W1031 23:08:28.124696 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 550 times. I1031 23:08:28.336268 5857 tablet_service.cc:691] Processing DeleteTablet for tablet 117cd74bcdab4942901acb0005cdd702 with delete_type TABLET_DATA_DELETED (Replaced by acb401e40e7049fbbfdbeb40474353d9 at 2017-10-31 23:04:52 CST) from {username='work'} at 10.108.187.10:53997 I1031 23:08:28.380982 5857 tablet_service.cc:691] Processing DeleteTablet for tablet bff7dab34ccd4b42842077e82106f6fb with delete_type TABLET_DATA_DELETED (Replaced by 005a48fb466c4a52896842dd00124755 at 2017-10-31 23:04:52 CST) from {username='work'} at 10.108.187.10:53997 I1031 23:08:28.395464 5857 tablet_service.cc:691] Processing DeleteTablet for tablet 56155a2adc24450d89e338cc955bfa18 with delete_type TABLET_DATA_DELETED (Replaced by 80ba7f33ec864c5f96c61932edb5cdc4 at 2017-10-31 23:04:52 CST) from {username='work'} at 10.108.187.10:53997 W1031 23:08:28.608939 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 491 times. W1031 23:08:28.624807 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 551 times. W1031 23:08:29.108978 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 492 times. W1031 23:08:29.124934 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 552 times. W1031 23:08:29.608974 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 493 times. W1031 23:08:29.625058 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 553 times. W1031 23:08:30.109165 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 494 times. W1031 23:08:30.125146 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 554 times. W1031 23:08:30.609179 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 495 times. W1031 23:08:30.625198 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 555 times. W1031 23:08:31.109266 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 496 times. W1031 23:08:31.125208 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 556 times. W1031 23:08:31.609247 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 497 times. W1031 23:08:31.625331 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 557 times. W1031 23:08:32.109416 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 498 times. W1031 23:08:32.125466 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 558 times. W1031 23:08:32.609505 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 499 times. W1031 23:08:32.625517 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 559 times. W1031 23:08:33.109550 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 500 times. W1031 23:08:33.125540 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 560 times. W1031 23:08:33.609659 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 501 times. W1031 23:08:33.625669 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 561 times. W1031 23:08:34.109712 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 502 times. W1031 23:08:34.125716 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 562 times. W1031 23:08:34.609767 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 503 times. W1031 23:08:34.625808 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 563 times. W1031 23:08:35.109844 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 504 times. W1031 23:08:35.125831 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 564 times. W1031 23:08:35.609952 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 505 times. W1031 23:08:35.625864 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 565 times. W1031 23:08:36.110043 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 506 times. W1031 23:08:36.126039 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 566 times. W1031 23:08:36.610146 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 507 times. W1031 23:08:36.626031 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 567 times. W1031 23:08:37.110159 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 508 times. W1031 23:08:37.126111 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 568 times. W1031 23:08:37.610195 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 509 times. W1031 23:08:37.626286 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 569 times. W1031 23:08:38.110369 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 510 times. W1031 23:08:38.126328 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 570 times. W1031 23:08:38.610469 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 511 times. W1031 23:08:38.626376 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 571 times. W1031 23:08:39.110528 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 512 times. W1031 23:08:39.126569 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 572 times. W1031 23:08:39.610651 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 513 times. W1031 23:08:39.626519 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 573 times. W1031 23:08:40.110713 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 514 times. W1031 23:08:40.126754 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 574 times. W1031 23:08:40.610708 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 515 times. W1031 23:08:40.626801 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 575 times. W1031 23:08:41.110853 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 516 times. W1031 23:08:41.126878 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 576 times. W1031 23:08:41.610951 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 517 times. W1031 23:08:41.626863 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 577 times. W1031 23:08:42.111057 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 518 times. W1031 23:08:42.127050 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 578 times. W1031 23:08:42.611129 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 519 times. W1031 23:08:42.626965 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 579 times. W1031 23:08:43.111233 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 520 times. W1031 23:08:43.127135 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 580 times. W1031 23:08:43.611299 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 521 times. W1031 23:08:43.627197 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 581 times. W1031 23:08:44.111328 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 522 times. W1031 23:08:44.127323 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 582 times. W1031 23:08:44.611459 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 523 times. W1031 23:08:44.627403 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 583 times. W1031 23:08:45.111574 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 524 times. W1031 23:08:45.127444 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 584 times. W1031 23:08:45.611553 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 525 times. W1031 23:08:45.627631 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 585 times. W1031 23:08:46.111663 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 526 times. W1031 23:08:46.127681 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 586 times. W1031 23:08:46.611624 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 527 times. W1031 23:08:46.627686 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 587 times. W1031 23:08:47.111778 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 528 times. W1031 23:08:47.127794 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 588 times. W1031 23:08:47.611799 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 529 times. W1031 23:08:47.627780 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 589 times. W1031 23:08:48.111917 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 530 times. W1031 23:08:48.127923 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 590 times. W1031 23:08:48.611974 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 531 times. W1031 23:08:48.628072 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 591 times. W1031 23:08:49.112109 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 532 times. W1031 23:08:49.128116 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 592 times. W1031 23:08:49.612193 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 533 times. W1031 23:08:49.628257 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 593 times. W1031 23:08:50.112275 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 534 times. W1031 23:08:50.128433 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 594 times. W1031 23:08:50.612411 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 535 times. W1031 23:08:50.628316 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 595 times. W1031 23:08:51.112520 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 536 times. W1031 23:08:51.128433 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 596 times. W1031 23:08:51.612532 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 537 times. W1031 23:08:51.628563 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 597 times. W1031 23:08:52.112589 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 538 times. W1031 23:08:52.128679 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 598 times. W1031 23:08:52.612702 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 539 times. W1031 23:08:52.628717 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 599 times. W1031 23:08:53.112849 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 540 times. W1031 23:08:53.128782 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 600 times. W1031 23:08:53.612854 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 541 times. W1031 23:08:53.628849 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 601 times. W1031 23:08:54.112959 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 542 times. W1031 23:08:54.129035 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 602 times. W1031 23:08:54.613075 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 543 times. W1031 23:08:54.629072 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 603 times. W1031 23:08:55.113103 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 544 times. W1031 23:08:55.129086 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 604 times. W1031 23:08:55.613227 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 545 times. W1031 23:08:55.629171 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 605 times. W1031 23:08:56.113327 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 546 times. W1031 23:08:56.129302 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 606 times. W1031 23:08:56.613421 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 547 times. W1031 23:08:56.629367 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 607 times. W1031 23:08:57.113535 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 548 times. W1031 23:08:57.129479 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 608 times. W1031 23:08:57.613601 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 549 times. W1031 23:08:57.629570 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 609 times. W1031 23:08:58.113637 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 550 times. W1031 23:08:58.129747 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 610 times. W1031 23:08:58.613770 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 551 times. W1031 23:08:58.629784 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 611 times. W1031 23:08:59.113843 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 552 times. W1031 23:08:59.129827 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 612 times. W1031 23:08:59.613973 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 553 times. W1031 23:08:59.629827 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 613 times. W1031 23:09:00.113992 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 554 times. W1031 23:09:00.129973 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 614 times. W1031 23:09:00.614112 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 555 times. W1031 23:09:00.630007 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 615 times. W1031 23:09:01.114122 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 556 times. W1031 23:09:01.130120 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 616 times. W1031 23:09:01.614142 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 557 times. W1031 23:09:01.630097 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 617 times. W1031 23:09:02.114290 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 558 times. W1031 23:09:02.130197 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 618 times. W1031 23:09:02.614320 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 559 times. W1031 23:09:02.630280 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 619 times. W1031 23:09:03.114433 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 560 times. W1031 23:09:03.130353 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 620 times. W1031 23:09:03.614536 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 561 times. W1031 23:09:03.630388 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 621 times. W1031 23:09:04.114624 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 562 times. W1031 23:09:04.130483 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 622 times. W1031 23:09:04.614668 5816 consensus_peers.cc:357] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet 991306ec7acb449c89e5dd29d90e8d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 563 times. W1031 23:09:04.630535 5816 consensus_peers.cc:357] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd -> Peer 23f962e4a1744381ad5fa0d2d8b10241 (c3-kudu-tst-st07.bj:18700): Couldn't send request to peer 23f962e4a1744381ad5fa0d2d8b10241 for tablet fcde65c4e4cf4df29b9ef9884ce292b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 623 times. I1031 23:09:04.740254 5884 raft_consensus.cc:1708] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate 23f962e4a1744381ad5fa0d2d8b10241 for term 1 because replica is either leader or believes a valid leader to be alive. I1031 23:09:05.114863 34581 consensus_queue.cc:695] T 991306ec7acb449c89e5dd29d90e8d55 P 0f53a0d3ef7e44ebb0365c800752d5bd [LEADER]: Connected to new peer: Peer: 23f962e4a1744381ad5fa0d2d8b10241, Is new: false, Last received: 0.0, Next index: 1, Last known committed idx: 0, Last exchange result: ERROR, Needs tablet copy: false I1031 23:09:05.134994 34138 consensus_queue.cc:695] T fcde65c4e4cf4df29b9ef9884ce292b2 P 0f53a0d3ef7e44ebb0365c800752d5bd [LEADER]: Connected to new peer: Peer: 23f962e4a1744381ad5fa0d2d8b10241, Is new: false, Last received: 0.0, Next index: 1, Last known committed idx: 0, Last exchange result: ERROR, Needs tablet copy: false I1031 23:09:10.302116 5734 tablet.cc:911] T eab9e68ae4f74574a6195e7738a310a0 P 0f53a0d3ef7e44ebb0365c800752d5bd: Flush: entering stage 1 (old memrowset already frozen for inserts) I1031 23:09:10.302165 5734 compaction.cc:877] Selected 1 rowsets to compact: I1031 23:09:10.302177 5734 compaction.cc:880] memrowset(current size on disk: ~0 bytes) I1031 23:09:10.302189 5734 tablet.cc:913] T eab9e68ae4f74574a6195e7738a310a0 P 0f53a0d3ef7e44ebb0365c800752d5bd: Memstore in-memory size: 3751 bytes I1031 23:09:10.302214 5734 tablet.cc:1174] T eab9e68ae4f74574a6195e7738a310a0 P 0f53a0d3ef7e44ebb0365c800752d5bd: Flush: entering phase 1 (flushing snapshot). Phase 1 snapshot: MvccSnapshot[committed={T|T < 6182757722233839616 or (T in {6182757722233839616})}] I1031 23:09:10.302351 5734 multi_column_writer.cc:85] Opened CFile writer for column ts[int32 NOT NULL] I1031 23:09:10.302496 5734 multi_column_writer.cc:85] Opened CFile writer for column id[int32 NOT NULL] I1031 23:09:10.553804 5734 tablet.cc:1259] T eab9e68ae4f74574a6195e7738a310a0 P 0f53a0d3ef7e44ebb0365c800752d5bd: Flush: entering phase 2 (starting to duplicate updates in new rowsets) I1031 23:09:10.786950 5734 tablet.cc:1314] T eab9e68ae4f74574a6195e7738a310a0 P 0f53a0d3ef7e44ebb0365c800752d5bd: Flush Phase 2: carrying over any updates which arrived during Phase 1 I1031 23:09:10.787017 5734 tablet.cc:1316] T eab9e68ae4f74574a6195e7738a310a0 P 0f53a0d3ef7e44ebb0365c800752d5bd: Phase 2 snapshot: MvccSnapshot[committed={T|T < 6182757722233839616 or (T in {6182757722233839616})}] I1031 23:09:11.322384 5734 tablet.cc:1358] T eab9e68ae4f74574a6195e7738a310a0 P 0f53a0d3ef7e44ebb0365c800752d5bd: Flush successful on 12 rows (4330 bytes) I1031 23:09:11.322479 5734 maintenance_manager.cc:419] Time spent running FlushMRSOp(eab9e68ae4f74574a6195e7738a310a0): real 1.254s user 0.826s sys 0.003s I1031 23:09:11.322645 5734 maintenance_manager.cc:425] P 0f53a0d3ef7e44ebb0365c800752d5bd: FlushMRSOp(eab9e68ae4f74574a6195e7738a310a0) metrics: {"cfile_init":1,"fdatasync":11,"fdatasync_us":401179,"lbm_read_time_us":8,"lbm_reads_lt_1ms":4,"lbm_write_time_us":156,"lbm_writes_lt_1ms":26,"tcmalloc_contention_cycles":49024} I1031 23:10:48.870229 5734 maintenance_manager.cc:419] Time spent running UndoDeltaBlockGCOp(eab9e68ae4f74574a6195e7738a310a0): real 0.226s user 0.148s sys 0.004s I1031 23:10:48.870582 5734 maintenance_manager.cc:425] P 0f53a0d3ef7e44ebb0365c800752d5bd: UndoDeltaBlockGCOp(eab9e68ae4f74574a6195e7738a310a0) metrics: {"cfile_init":2,"data dir 5.queue_time_us":228,"data dir 5.run_cpu_time_us":66,"data dir 5.run_wall_time_us":66,"fdatasync":1,"fdatasync_us":50515,"lbm_read_time_us":19,"lbm_reads_lt_1ms":8,"thread_start_us":215,"threads_started":1} I1031 23:10:49.253904 5734 maintenance_manager.cc:419] Time spent running UndoDeltaBlockGCOp(cd5d028ce30a4eb19eacbecbf5a3486b): real 0.207s user 0.148s sys 0.003s I1031 23:10:49.254245 5734 maintenance_manager.cc:425] P 0f53a0d3ef7e44ebb0365c800752d5bd: UndoDeltaBlockGCOp(cd5d028ce30a4eb19eacbecbf5a3486b) metrics: {"cfile_init":1,"data dir 0.queue_time_us":768,"data dir 0.run_cpu_time_us":55,"data dir 0.run_wall_time_us":55,"fdatasync":1,"fdatasync_us":39560,"lbm_read_time_us":12,"lbm_reads_lt_1ms":4,"tcmalloc_contention_cycles":85888,"thread_start_us":753,"threads_started":1} I1031 23:10:49.681918 5734 tablet.cc:911] T 4b7b2e29f6a44c6687ca91d407a725d4 P 0f53a0d3ef7e44ebb0365c800752d5bd: Flush: entering stage 1 (old memrowset already frozen for inserts) I1031 23:10:49.681957 5734 compaction.cc:877] Selected 1 rowsets to compact: I1031 23:10:49.681969 5734 compaction.cc:880] memrowset(current size on disk: ~0 bytes) I1031 23:10:49.681980 5734 tablet.cc:913] T 4b7b2e29f6a44c6687ca91d407a725d4 P 0f53a0d3ef7e44ebb0365c800752d5bd: Memstore in-memory size: 3751 bytes I1031 23:10:49.682003 5734 tablet.cc:1174] T 4b7b2e29f6a44c6687ca91d407a725d4 P 0f53a0d3ef7e44ebb0365c800752d5bd: Flush: entering phase 1 (flushing snapshot). Phase 1 snapshot: MvccSnapshot[committed={T|T < 6182758950407372800 or (T in {6182758950407372800})}] I1031 23:10:49.682142 5734 multi_column_writer.cc:85] Opened CFile writer for column ts[int32 NOT NULL] I1031 23:10:49.682183 5734 multi_column_writer.cc:85] Opened CFile writer for column id[int32 NOT NULL] I1031 23:10:49.829530 5734 tablet.cc:1259] T 4b7b2e29f6a44c6687ca91d407a725d4 P 0f53a0d3ef7e44ebb0365c800752d5bd: Flush: entering phase 2 (starting to duplicate updates in new rowsets) I1031 23:10:50.057235 5734 tablet.cc:1314] T 4b7b2e29f6a44c6687ca91d407a725d4 P 0f53a0d3ef7e44ebb0365c800752d5bd: Flush Phase 2: carrying over any updates which arrived during Phase 1 I1031 23:10:50.057313 5734 tablet.cc:1316] T 4b7b2e29f6a44c6687ca91d407a725d4 P 0f53a0d3ef7e44ebb0365c800752d5bd: Phase 2 snapshot: MvccSnapshot[committed={T|T < 6182758950407372800 or (T in {6182758950407372800})}] I1031 23:10:50.460490 5734 tablet.cc:1358] T 4b7b2e29f6a44c6687ca91d407a725d4 P 0f53a0d3ef7e44ebb0365c800752d5bd: Flush successful on 11 rows (4330 bytes) I1031 23:10:50.460597 5734 maintenance_manager.cc:419] Time spent running FlushMRSOp(4b7b2e29f6a44c6687ca91d407a725d4): real 1.011s user 0.808s sys 0.005s I1031 23:10:50.460752 5734 maintenance_manager.cc:425] P 0f53a0d3ef7e44ebb0365c800752d5bd: FlushMRSOp(4b7b2e29f6a44c6687ca91d407a725d4) metrics: {"cfile_init":1,"fdatasync":11,"fdatasync_us":182227,"lbm_read_time_us":6,"lbm_reads_lt_1ms":4,"lbm_write_time_us":159,"lbm_writes_lt_1ms":26,"tcmalloc_contention_cycles":140672} W1031 23:11:32.329298 24525 log.cc:565] Time spent T b0174e1a99694688bdc932c6a00233ce P 0f53a0d3ef7e44ebb0365c800752d5bd: Append to log took a long time: real 0.104s user 0.000s sys 0.000s I1031 23:12:31.994580 5734 tablet.cc:911] T ee1d4d81f7b944c3884ec0ac9aa1373f P 0f53a0d3ef7e44ebb0365c800752d5bd: Flush: entering stage 1 (old memrowset already frozen for inserts) I1031 23:12:31.994617 5734 compaction.cc:877] Selected 1 rowsets to compact: I1031 23:12:31.994639 5734 compaction.cc:880] memrowset(current size on disk: ~0 bytes) I1031 23:12:31.994652 5734 tablet.cc:913] T ee1d4d81f7b944c3884ec0ac9aa1373f P 0f53a0d3ef7e44ebb0365c800752d5bd: Memstore in-memory size: 3751 bytes I1031 23:12:31.994673 5734 tablet.cc:1174] T ee1d4d81f7b944c3884ec0ac9aa1373f P 0f53a0d3ef7e44ebb0365c800752d5bd: Flush: entering phase 1 (flushing snapshot). Phase 1 snapshot: MvccSnapshot[committed={T|T < 6182759347498897410 or (T in {6182759347498897410})}] I1031 23:12:31.994873 5734 multi_column_writer.cc:85] Opened CFile writer for column ts[int32 NOT NULL] I1031 23:12:31.994926 5734 multi_column_writer.cc:85] Opened CFile writer for column id[int32 NOT NULL] I1031 23:12:32.102428 5734 tablet.cc:1259] T ee1d4d81f7b944c3884ec0ac9aa1373f P 0f53a0d3ef7e44ebb0365c800752d5bd: Flush: entering phase 2 (starting to duplicate updates in new rowsets) I1031 23:12:32.331190 5734 tablet.cc:1314] T ee1d4d81f7b944c3884ec0ac9aa1373f P 0f53a0d3ef7e44ebb0365c800752d5bd: Flush Phase 2: carrying over any updates which arrived during Phase 1 I1031 23:12:32.331254 5734 tablet.cc:1316] T ee1d4d81f7b944c3884ec0ac9aa1373f P 0f53a0d3ef7e44ebb0365c800752d5bd: Phase 2 snapshot: MvccSnapshot[committed={T|T < 6182759347498897410 or (T in {6182759347498897410})}] I1031 23:12:32.746981 5734 tablet.cc:1358] T ee1d4d81f7b944c3884ec0ac9aa1373f P 0f53a0d3ef7e44ebb0365c800752d5bd: Flush successful on 15 rows (4330 bytes) I1031 23:12:32.747076 5734 maintenance_manager.cc:419] Time spent running FlushMRSOp(ee1d4d81f7b944c3884ec0ac9aa1373f): real 0.981s user 0.812s sys 0.004s I1031 23:12:32.747242 5734 maintenance_manager.cc:425] P 0f53a0d3ef7e44ebb0365c800752d5bd: FlushMRSOp(ee1d4d81f7b944c3884ec0ac9aa1373f) metrics: {"cfile_init":1,"fdatasync":11,"fdatasync_us":148779,"lbm_read_time_us":8,"lbm_reads_lt_1ms":4,"lbm_write_time_us":219,"lbm_writes_lt_1ms":26,"tcmalloc_contention_cycles":67712} I1031 23:14:09.480738 5734 maintenance_manager.cc:419] Time spent running UndoDeltaBlockGCOp(88bb1096bf634bf48c669a680936e511): real 0.220s user 0.149s sys 0.003s I1031 23:14:09.481101 5734 maintenance_manager.cc:425] P 0f53a0d3ef7e44ebb0365c800752d5bd: UndoDeltaBlockGCOp(88bb1096bf634bf48c669a680936e511) metrics: {"data dir 4.queue_time_us":307,"data dir 4.run_cpu_time_us":57,"data dir 4.run_wall_time_us":57,"fdatasync":1,"fdatasync_us":52438,"thread_start_us":286,"threads_started":1} I1031 23:14:09.874125 5734 maintenance_manager.cc:419] Time spent running UndoDeltaBlockGCOp(8b235ddfb2cf4474a8649365d4a75ad7): real 0.210s user 0.150s sys 0.003s I1031 23:14:09.874212 5734 maintenance_manager.cc:425] P 0f53a0d3ef7e44ebb0365c800752d5bd: UndoDeltaBlockGCOp(8b235ddfb2cf4474a8649365d4a75ad7) metrics: {"data dir 6.queue_time_us":228,"data dir 6.run_cpu_time_us":49,"data dir 6.run_wall_time_us":50,"fdatasync":1,"fdatasync_us":42189,"tcmalloc_contention_cycles":29696,"thread_start_us":214,"threads_started":1} I1031 23:14:10.274343 5734 maintenance_manager.cc:419] Time spent running UndoDeltaBlockGCOp(8fa7cc8c413f452fb2c45eb13d186632): real 0.207s user 0.152s sys 0.002s I1031 23:14:10.274658 5734 maintenance_manager.cc:425] P 0f53a0d3ef7e44ebb0365c800752d5bd: UndoDeltaBlockGCOp(8fa7cc8c413f452fb2c45eb13d186632) metrics: {"data dir 2.queue_time_us":742,"data dir 2.run_cpu_time_us":51,"data dir 2.run_wall_time_us":51,"fdatasync":1,"fdatasync_us":37165,"thread_start_us":728,"threads_started":1} I1031 23:14:10.474387 5734 delta_tracker.cc:686] T ee1d4d81f7b944c3884ec0ac9aa1373f P 0f53a0d3ef7e44ebb0365c800752d5bd: Flushing 2 deltas from DMS 1... I1031 23:14:10.510134 5734 delta_tracker.cc:628] T ee1d4d81f7b944c3884ec0ac9aa1373f P 0f53a0d3ef7e44ebb0365c800752d5bd: Flushed delta block: 0000000044990300 ts range: [6182759443259944960, 6182759443344883712] I1031 23:14:10.510175 5734 delta_tracker.cc:641] T ee1d4d81f7b944c3884ec0ac9aa1373f P 0f53a0d3ef7e44ebb0365c800752d5bd: Reopened delta block for read: 0000000044990300 I1031 23:14:10.695263 5734 maintenance_manager.cc:419] Time spent running FlushDeltaMemStoresOp(ee1d4d81f7b944c3884ec0ac9aa1373f): real 0.222s user 0.120s sys 0.003s I1031 23:14:10.695340 5734 maintenance_manager.cc:425] P 0f53a0d3ef7e44ebb0365c800752d5bd: FlushDeltaMemStoresOp(ee1d4d81f7b944c3884ec0ac9aa1373f) metrics: {"fdatasync":3,"fdatasync_us":84184,"lbm_write_time_us":65,"lbm_writes_lt_1ms":6,"tcmalloc_contention_cycles":592128} I1031 23:14:11.288409 5734 tablet.cc:911] T f692b82bfeb54947bf120ff1f56ff38b P 0f53a0d3ef7e44ebb0365c800752d5bd: Flush: entering stage 1 (old memrowset already frozen for inserts) I1031 23:14:11.288452 5734 compaction.cc:877] Selected 1 rowsets to compact: I1031 23:14:11.288470 5734 compaction.cc:880] memrowset(current size on disk: ~0 bytes) I1031 23:14:11.288485 5734 tablet.cc:913] T f692b82bfeb54947bf120ff1f56ff38b P 0f53a0d3ef7e44ebb0365c800752d5bd: Memstore in-memory size: 3751 bytes I1031 23:14:11.288513 5734 tablet.cc:1174] T f692b82bfeb54947bf120ff1f56ff38b P 0f53a0d3ef7e44ebb0365c800752d5bd: Flush: entering phase 1 (flushing snapshot). Phase 1 snapshot: MvccSnapshot[committed={T|T < 6182759838853062656 or (T in {6182759838853062656})}] I1031 23:14:11.288664 5734 multi_column_writer.cc:85] Opened CFile writer for column ts[int32 NOT NULL] I1031 23:14:11.288714 5734 multi_column_writer.cc:85] Opened CFile writer for column id[int32 NOT NULL] I1031 23:14:11.425284 5734 tablet.cc:1259] T f692b82bfeb54947bf120ff1f56ff38b P 0f53a0d3ef7e44ebb0365c800752d5bd: Flush: entering phase 2 (starting to duplicate updates in new rowsets) I1031 23:14:11.798666 5734 tablet.cc:1314] T f692b82bfeb54947bf120ff1f56ff38b P 0f53a0d3ef7e44ebb0365c800752d5bd: Flush Phase 2: carrying over any updates which arrived during Phase 1 I1031 23:14:11.798732 5734 tablet.cc:1316] T f692b82bfeb54947bf120ff1f56ff38b P 0f53a0d3ef7e44ebb0365c800752d5bd: Phase 2 snapshot: MvccSnapshot[committed={T|T < 6182759838853062656 or (T in {6182759838853062656})}] I1031 23:14:12.411051 5734 tablet.cc:1358] T f692b82bfeb54947bf120ff1f56ff38b P 0f53a0d3ef7e44ebb0365c800752d5bd: Flush successful on 12 rows (4374 bytes) I1031 23:14:12.411152 5734 maintenance_manager.cc:419] Time spent running FlushMRSOp(f692b82bfeb54947bf120ff1f56ff38b): real 1.497s user 1.283s sys 0.007s I1031 23:14:12.411315 5734 maintenance_manager.cc:425] P 0f53a0d3ef7e44ebb0365c800752d5bd: FlushMRSOp(f692b82bfeb54947bf120ff1f56ff38b) metrics: {"cfile_init":1,"fdatasync":11,"fdatasync_us":191444,"lbm_read_time_us":8,"lbm_reads_lt_1ms":4,"lbm_write_time_us":166,"lbm_writes_lt_1ms":26,"tcmalloc_contention_cycles":61952} I1031 23:17:47.504855 5734 maintenance_manager.cc:419] Time spent running UndoDeltaBlockGCOp(3409046e569643c3a083c3f27cfe1582): real 0.202s user 0.147s sys 0.003s I1031 23:17:47.505206 5734 maintenance_manager.cc:425] P 0f53a0d3ef7e44ebb0365c800752d5bd: UndoDeltaBlockGCOp(3409046e569643c3a083c3f27cfe1582) metrics: {"data dir 0.queue_time_us":218,"data dir 0.run_cpu_time_us":67,"data dir 0.run_wall_time_us":67,"fdatasync":1,"fdatasync_us":36573,"tcmalloc_contention_cycles":25728,"thread_start_us":204,"threads_started":1} I1031 23:17:47.698679 5734 tablet.cc:911] T 71170c928d684672bdc09e384fa7382d P 0f53a0d3ef7e44ebb0365c800752d5bd: Flush: entering stage 1 (old memrowset already frozen for inserts) I1031 23:17:47.698721 5734 compaction.cc:877] Selected 1 rowsets to compact: I1031 23:17:47.698734 5734 compaction.cc:880] memrowset(current size on disk: ~0 bytes) I1031 23:17:47.698740 5734 tablet.cc:913] T 71170c928d684672bdc09e384fa7382d P 0f53a0d3ef7e44ebb0365c800752d5bd: Memstore in-memory size: 4079383 bytes I1031 23:17:47.698750 5734 tablet.cc:1174] T 71170c928d684672bdc09e384fa7382d P 0f53a0d3ef7e44ebb0365c800752d5bd: Flush: entering phase 1 (flushing snapshot). Phase 1 snapshot: MvccSnapshot[committed={T|T < 6182760724161957888 or (T in {6182760724161957888})}] I1031 23:17:47.698865 5734 multi_column_writer.cc:85] Opened CFile writer for column olap_date[int32 NOT NULL] I1031 23:17:47.698912 5734 multi_column_writer.cc:85] Opened CFile writer for column timestamp[int64 NOT NULL] I1031 23:17:47.698945 5734 multi_column_writer.cc:85] Opened CFile writer for column tweet_id[int64 NOT NULL] I1031 23:17:47.698984 5734 multi_column_writer.cc:85] Opened CFile writer for column created_at[string NULLABLE] I1031 23:17:47.699023 5734 multi_column_writer.cc:85] Opened CFile writer for column text[string NULLABLE] I1031 23:17:47.699061 5734 multi_column_writer.cc:85] Opened CFile writer for column source[string NULLABLE] I1031 23:17:47.699097 5734 multi_column_writer.cc:85] Opened CFile writer for column user_id[int64 NULLABLE] I1031 23:17:47.699141 5734 multi_column_writer.cc:85] Opened CFile writer for column user_name[string NULLABLE] I1031 23:17:47.699182 5734 multi_column_writer.cc:85] Opened CFile writer for column user_location[string NULLABLE] I1031 23:17:47.699214 5734 multi_column_writer.cc:85] Opened CFile writer for column user_followers_count[int32 NULLABLE] I1031 23:17:47.699245 5734 multi_column_writer.cc:85] Opened CFile writer for column user_friends_count[int32 NULLABLE] I1031 23:17:48.064556 5734 tablet.cc:1259] T 71170c928d684672bdc09e384fa7382d P 0f53a0d3ef7e44ebb0365c800752d5bd: Flush: entering phase 2 (starting to duplicate updates in new rowsets) I1031 23:17:48.064827 5734 tablet.cc:1314] T 71170c928d684672bdc09e384fa7382d P 0f53a0d3ef7e44ebb0365c800752d5bd: Flush Phase 2: carrying over any updates which arrived during Phase 1 I1031 23:17:48.064843 5734 tablet.cc:1316] T 71170c928d684672bdc09e384fa7382d P 0f53a0d3ef7e44ebb0365c800752d5bd: Phase 2 snapshot: MvccSnapshot[committed={T|T < 6182760724161957888 or (T in {6182760724161957888})}] I1031 23:17:48.115191 5734 tablet.cc:1358] T 71170c928d684672bdc09e384fa7382d P 0f53a0d3ef7e44ebb0365c800752d5bd: Flush successful on 8771 rows (1486862 bytes) I1031 23:17:48.115442 5734 maintenance_manager.cc:419] Time spent running FlushMRSOp(71170c928d684672bdc09e384fa7382d): real 0.417s user 0.038s sys 0.006s I1031 23:17:48.115485 5734 maintenance_manager.cc:425] P 0f53a0d3ef7e44ebb0365c800752d5bd: FlushMRSOp(71170c928d684672bdc09e384fa7382d) metrics: {"cfile_init":1,"data dir 8.queue_time_us":796,"data dir 8.run_cpu_time_us":130,"data dir 8.run_wall_time_us":36017,"fdatasync":29,"fdatasync_us":355944,"lbm_read_time_us":7,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1954,"lbm_writes_lt_1ms":118,"thread_start_us":777,"threads_started":1} I1031 23:17:48.352066 5734 tablet.cc:911] T b0174e1a99694688bdc932c6a00233ce P 0f53a0d3ef7e44ebb0365c800752d5bd: Flush: entering stage 1 (old memrowset already frozen for inserts) I1031 23:17:48.352103 5734 compaction.cc:877] Selected 1 rowsets to compact: I1031 23:17:48.352109 5734 compaction.cc:880] memrowset(current size on disk: ~0 bytes) I1031 23:17:48.352115 5734 tablet.cc:913] T b0174e1a99694688bdc932c6a00233ce P 0f53a0d3ef7e44ebb0365c800752d5bd: Memstore in-memory size: 4079383 bytes I1031 23:17:48.352134 5734 tablet.cc:1174] T b0174e1a99694688bdc932c6a00233ce P 0f53a0d3ef7e44ebb0365c800752d5bd: Flush: entering phase 1 (flushing snapshot). Phase 1 snapshot: MvccSnapshot[committed={T|T < 6182760727608262656 or (T in {6182760727608262656})}] I1031 23:17:48.352278 5734 multi_column_writer.cc:85] Opened CFile writer for column olap_date[int32 NOT NULL] I1031 23:17:48.352321 5734 multi_column_writer.cc:85] Opened CFile writer for column timestamp[int64 NOT NULL] I1031 23:17:48.352361 5734 multi_column_writer.cc:85] Opened CFile writer for column tweet_id[int64 NOT NULL] I1031 23:17:48.352432 5734 multi_column_writer.cc:85] Opened CFile writer for column created_at[string NULLABLE] I1031 23:17:48.352470 5734 multi_column_writer.cc:85] Opened CFile writer for column text[string NULLABLE] I1031 23:17:48.352520 5734 multi_column_writer.cc:85] Opened CFile writer for column source[string NULLABLE] I1031 23:17:48.352563 5734 multi_column_writer.cc:85] Opened CFile writer for column user_id[int64 NULLABLE] I1031 23:17:48.352597 5734 multi_column_writer.cc:85] Opened CFile writer for column user_name[string NULLABLE] I1031 23:17:48.352641 5734 multi_column_writer.cc:85] Opened CFile writer for column user_location[string NULLABLE] I1031 23:17:48.352686 5734 multi_column_writer.cc:85] Opened CFile writer for column user_followers_count[int32 NULLABLE] I1031 23:17:48.352713 5734 multi_column_writer.cc:85] Opened CFile writer for column user_friends_count[int32 NULLABLE] I1031 23:17:48.623841 5734 tablet.cc:1259] T b0174e1a99694688bdc932c6a00233ce P 0f53a0d3ef7e44ebb0365c800752d5bd: Flush: entering phase 2 (starting to duplicate updates in new rowsets) I1031 23:17:48.624109 5734 tablet.cc:1314] T b0174e1a99694688bdc932c6a00233ce P 0f53a0d3ef7e44ebb0365c800752d5bd: Flush Phase 2: carrying over any updates which arrived during Phase 1 I1031 23:17:48.624135 5734 tablet.cc:1316] T b0174e1a99694688bdc932c6a00233ce P 0f53a0d3ef7e44ebb0365c800752d5bd: Phase 2 snapshot: MvccSnapshot[committed={T|T < 6182760728747966464 or (T in {6182760728747966464})}] I1031 23:17:48.673918 5734 tablet.cc:1358] T b0174e1a99694688bdc932c6a00233ce P 0f53a0d3ef7e44ebb0365c800752d5bd: Flush successful on 8898 rows (1520125 bytes) I1031 23:17:48.674181 5734 maintenance_manager.cc:419] Time spent running FlushMRSOp(b0174e1a99694688bdc932c6a00233ce): real 0.322s user 0.039s sys 0.003s I1031 23:17:48.674221 5734 maintenance_manager.cc:425] P 0f53a0d3ef7e44ebb0365c800752d5bd: FlushMRSOp(b0174e1a99694688bdc932c6a00233ce) metrics: {"cfile_init":1,"data dir 5.queue_time_us":227,"data dir 5.run_cpu_time_us":104,"data dir 5.run_wall_time_us":30419,"fdatasync":29,"fdatasync_us":263182,"lbm_read_time_us":8,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1989,"lbm_writes_lt_1ms":120,"thread_start_us":213,"threads_started":1} I1031 23:17:49.235062 5734 tablet.cc:911] T 867ed73769f842bda8756be0b016a15a P 0f53a0d3ef7e44ebb0365c800752d5bd: Flush: entering stage 1 (old memrowset already frozen for inserts) I1031 23:17:49.235107 5734 compaction.cc:877] Selected 1 rowsets to compact: I1031 23:17:49.235129 5734 compaction.cc:880] memrowset(current size on disk: ~0 bytes) I1031 23:17:49.235146 5734 tablet.cc:913] T 867ed73769f842bda8756be0b016a15a P 0f53a0d3ef7e44ebb0365c800752d5bd: Memstore in-memory size: 3751 bytes I1031 23:17:49.235183 5734 tablet.cc:1174] T 867ed73769f842bda8756be0b016a15a P 0f53a0d3ef7e44ebb0365c800752d5bd: Flush: entering phase 1 (flushing snapshot). Phase 1 snapshot: MvccSnapshot[committed={T|T < 6182760427256692736 or (T in {6182760427256692736})}] I1031 23:17:49.235337 5734 multi_column_writer.cc:85] Opened CFile writer for column ts[int32 NOT NULL] I1031 23:17:49.235469 5734 multi_column_writer.cc:85] Opened CFile writer for column id[int32 NOT NULL] I1031 23:17:49.361093 5734 tablet.cc:1259] T 867ed73769f842bda8756be0b016a15a P 0f53a0d3ef7e44ebb0365c800752d5bd: Flush: entering phase 2 (starting to du